Re: logical decoding : exceeded maxAllocatedDescs for .spill files - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: logical decoding : exceeded maxAllocatedDescs for .spill files
Date
Msg-id CAA4eK1JUa1kVSW-NPg4hqycZZF3j+i_z9h4R4hcS7FKOx_OXxA@mail.gmail.com
Whole thread Raw
In response to Re: logical decoding : exceeded maxAllocatedDescs for .spill files  (Amit Khandekar <amitdkhan.pg@gmail.com>)
List pgsql-hackers
On Wed, Nov 20, 2019 at 2:18 PM Amit Khandekar <amitdkhan.pg@gmail.com> wrote:
>
> On Wed, 20 Nov 2019 at 13:10, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > See comment in pgunlink() "We need to loop because even though
> > PostgreSQL uses flags that allow unlink while the file is open, other
> > applications might have the file
> > open without those flags.".  Can you once see if there is any flag
> > that you have missed to pass to allow this?
>
> > If there is nothing we
> > can do about it, then we might need to use some different API or maybe
> > define a new API that can handle this.
>
> There were objections against modifying the vfd api only for this
> replication-related use-case. Having a new API will require all the
> changes required to enable the virtual FDs feature that we need from
> vfd. If nothing works out from the FILE_SHARE_DELETE thing,
>

While experimenting with FILE_SHARE_DELETE, I think you can once try
to open/close the file before unlink.  If you read the specs [1] of
this flag, it seems they allow you to open the file with delete access
even when it is already opened by someone else.  I am not sure if that
is helpful, but at least we can try out.

> I am
> thinking, we can use VFD, plus we can keep track of per-subtransaction
> vfd handles, and do something similar to AtEOSubXact_Files().
>

I think if we can't make the current API work, then it is better to
sketch the design for this approach and probably the design of new API
using existing infrastructure.  Then we can see which approach people
prefer.


[1] - Enables subsequent open operations on a file or device to
request delete access.  Otherwise, other processes cannot open the
file or device if they request delete access.
(https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea)


-- 
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com



pgsql-hackers by date:

Previous
From: "Smith, Peter"
Date:
Subject: RE: Proposal: Make use of C99 designated initialisers fornulls/values arrays
Next
From: Dilip Kumar
Date:
Subject: Re: [HACKERS] Block level parallel vacuum