Thread: Re: Windows pg_basebackup unable to create >2GB pg_wal.tar tarballs ("could not close file: Invalid argument" when creating pg_wal.tar of size ~ 2^31 bytes)

On Thu, Nov 21, 2024 at 11:44 PM Jakub Wartak
<jakub.wartak@enterprisedb.com> wrote:
> This literally looks like something like off_t/size_t would be limited to 2^31 somewhere.

off_t is 32 bits on Windows.  I'd be quite suspicious of the
arithmetic involving 'currpos'.  What happens if you change all the
off_t in walmethods.c/.h to pgoff_t?  The lseek() is suspicious too,
and might need to be redirected to _lseeki64().

On Thu, Nov 21, 2024 at 5:53 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> The lseek() is suspicious too,
> and might need to be redirected to _lseeki64().

There's a call to ftruncate() in there too. Looks like its Windows
definition is also 32-bit:

    #define ftruncate(a,b) chsize(a,b)

--Jacob

On Fri, Nov 22, 2024 at 8:59 AM Jacob Champion
<jacob.champion@enterprisedb.com> wrote:
> On Thu, Nov 21, 2024 at 5:53 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> > The lseek() is suspicious too,
> > and might need to be redirected to _lseeki64().
>
> There's a call to ftruncate() in there too. Looks like its Windows
> definition is also 32-bit:
>
>     #define ftruncate(a,b) chsize(a,b)

Ah, yes.  I wrote a draft fix for those two and more in the 0002
patch[1] for another project[2].

(I'm still planning on proposing the main idea in that thread --
allowing large files for relations -- but I was talked out of doing a
generalised pgoff_t-ification drilling through loads of layers all
over our tree just for Windows, so the idea was that Windows would
simply not get the large file support, and I dropped that patch from
that project.  But you can see the functions with that problem that I
found, and how I thought we should fix them. Obviously I didn't know
we still had more bugs around existing ways to make large files, which
various people have tackled over the years, so there are some
functions that are 64 bit capable like stat, just not the full set...)

As for "could not close ...", I guess dodgy data type maths must be
causing tar_close() to return -1 while trying to compute padding or
something like that, it's not literally close() that is failing.

[1]
https://www.postgresql.org/message-id/attachment/146666/0002-Use-pgoff_t-in-system-call-replacements-on-Windows.patch
[2] https://www.postgresql.org/message-id/CA%2BhUKG%2BBGXwMbrvzXAjL8VMGf25y_ga_XnO741g10y0%3Dm6dDiA%40mail.gmail.com