Thread: Re: pg_restore [archiver] file offset in dump file is too
> > Andrew Dunstan <andrew@dunslane.net> writes: > >> Hmm. Windows reports an offset size of 4 bytes on a dump I > just made > >> ... is that relevant? What governs it? > > > > [ looks again... ] Hm, that is a 40Gb dump Kevin is > complaining of, > > isn't it. Do our Windows builds have LARGEFILE support? > > > > I think from a cursory investigation the short answer is no, > but they probably could. If so, that should definitely go on > the TODO list. Windows gurus, any thoughts? Windows certainly supports large files. I don't see why we wouldn't pick this up in autoconf, perhaps the mingw libraries don't? Definitly worth investigating, no time for 8.1, so putting it on TODO seems very good :-) //Magnus
Magnus Hagander wrote: >>>Andrew Dunstan <andrew@dunslane.net> writes: >>> >>> >>>>Hmm. Windows reports an offset size of 4 bytes on a dump I >>>> >>>> >>just made >> >> >>>>... is that relevant? What governs it? >>>> >>>> >>>[ looks again... ] Hm, that is a 40Gb dump Kevin is >>> >>> >>complaining of, >> >> >>>isn't it. Do our Windows builds have LARGEFILE support? >>> >>> >>> >>I think from a cursory investigation the short answer is no, >>but they probably could. If so, that should definitely go on >>the TODO list. Windows gurus, any thoughts? >> >> > >Windows certainly supports large files. I don't see why we wouldn't pick >this up in autoconf, perhaps the mingw libraries don't? >Definitly worth investigating, no time for 8.1, so putting it on TODO >seems very good :-) > > There is no fseeko in the Windows libraries, nor any provision in the mingw headers that I can see for a 64 bit off_t. So we would need to roll our own to some extent - I think we need more than just a bit of configure cleverness. However, there is a Windows library routine to do a 64bit seek and return the file position, so we could fairly easily implement fseeko and ftello based on that. See http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes: > There is no fseeko in the Windows libraries, nor any provision in the > mingw headers that I can see for a 64 bit off_t. So we would need to > roll our own to some extent - I think we need more than just a bit of > configure cleverness. > However, there is a Windows library routine to do a 64bit seek and > return the file position, so we could fairly easily implement fseeko and > ftello based on that. See > http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp Is there any risk that the mingw libraries would fail when manipulating a file whose current offset exceeds 32 bits? I'm wondering if we'd have to roll our own stdio in toto, not just fseeko/ftello :-( regards, tom lane
Tom Lane wrote: >Andrew Dunstan <andrew@dunslane.net> writes: > > >>There is no fseeko in the Windows libraries, nor any provision in the >>mingw headers that I can see for a 64 bit off_t. So we would need to >>roll our own to some extent - I think we need more than just a bit of >>configure cleverness. >> >> > > > >>However, there is a Windows library routine to do a 64bit seek and >>return the file position, so we could fairly easily implement fseeko and >>ftello based on that. See >>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp >> >> > >Is there any risk that the mingw libraries would fail when manipulating >a file whose current offset exceeds 32 bits? I'm wondering if we'd have >to roll our own stdio in toto, not just fseeko/ftello :-( > > > > AFAIK all this is coming straight from the base Windows libraries. I guess we'll have to test it when we get around to it. cheers andrew
Andrew Dunstan wrote: > There is no fseeko in the Windows libraries, nor any provision in the > mingw headers that I can see for a 64 bit off_t. So we would need to > roll our own to some extent - I think we need more than just a bit of > configure cleverness. > > However, there is a Windows library routine to do a 64bit seek and > return the file position, so we could fairly easily implement fseeko and > ftello based on that. See > http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp See src/port/fseeko.c for a version built on fsetpos(). -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: >Andrew Dunstan wrote: > > >>There is no fseeko in the Windows libraries, nor any provision in the >>mingw headers that I can see for a 64 bit off_t. So we would need to >>roll our own to some extent - I think we need more than just a bit of >>configure cleverness. >> >>However, there is a Windows library routine to do a 64bit seek and >>return the file position, so we could fairly easily implement fseeko and >>ftello based on that. See >>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp >> >> > >See src/port/fseeko.c for a version built on fsetpos(). > > > Yeah, I'm not made warm and fuzzy by the comments about fpos_t in mingw's stdio.h, though: /** An opaque data type used for storing file positions... The contents of* this type are unknown, but we (the compiler)need to know the size* because the programmer using fgetpos and fsetpos will be setting aside* storage for fpos_tstructres. Actually I tested using a byte array and* it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).*Perhaps an unsigned long? TODO? It's definitely a 64-bit number in* MSVCRT however, and for now `long long'will do.*/ But the example program on MSDN contains "pos = 14", which leads one to assume that it really is some simple int underneath. cheers andrew cheers andrew
Added to TODO:
o Add long file support for binary pg_dump output
While Win32 supports 64-bit files, the MinGW API does not, meaning we have to build an fseeko
replacementon top of the Win32 API, and we have to make sure MinGW handles it.
---------------------------------------------------------------------------
Andrew Dunstan wrote:
>
>
> Bruce Momjian wrote:
>
> >Andrew Dunstan wrote:
> >
> >
> >>There is no fseeko in the Windows libraries, nor any provision in the
> >>mingw headers that I can see for a 64 bit off_t. So we would need to
> >>roll our own to some extent - I think we need more than just a bit of
> >>configure cleverness.
> >>
> >>However, there is a Windows library routine to do a 64bit seek and
> >>return the file position, so we could fairly easily implement fseeko and
> >>ftello based on that. See
> >>http://msdn.microsoft.com/library/en-us/vclib/html/_crt__lseek.2c_._lseeki64.asp
> >>
> >>
> >
> >See src/port/fseeko.c for a version built on fsetpos().
> >
> >
> >
>
> Yeah, I'm not made warm and fuzzy by the comments about fpos_t in
> mingw's stdio.h, though:
>
> /*
> * An opaque data type used for storing file positions... The contents of
> * this type are unknown, but we (the compiler) need to know the size
> * because the programmer using fgetpos and fsetpos will be setting aside
> * storage for fpos_t structres. Actually I tested using a byte array and
> * it is fairly evident that the fpos_t type is a long (in CRTDLL.DLL).
> * Perhaps an unsigned long? TODO? It's definitely a 64-bit number in
> * MSVCRT however, and for now `long long' will do.
> */
>
>
> But the example program on MSDN contains "pos = 14", which leads one to
> assume that it really is some simple int underneath.
>
> cheers
>
> andrew
>
> cheers
>
> andrew
>
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square,
Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Added to TODO:
> o Add long file support for binary pg_dump output
> While Win32 supports 64-bit files, the MinGW API does not,
> meaning we have to build an fseeko replacement on top of the
> Win32 API, and we have to make sure MinGW handles it.
Wouldn't it be better to lobby the MinGW folk to fix their problem?
Or even help them with it? I can't see the rationale for implementing
a workaround that helps only us.
regards, tom lane
Tom Lane wrote: >> While Win32 supports 64-bit files, the MinGW API does not, >> meaning we have to build an fseeko replacement on top of the >> Win32 API, and we have to make sure MinGW handles it. >> >> > >Wouldn't it be better to lobby the MinGW folk to fix their problem? >Or even help them with it? I can't see the rationale for implementing >a workaround that helps only us. > > > > There is a library available from the gnuwin32 project that advertises fseeko and fseeko64. So we probably have a choice of requiring this library or doing it ourselves. see http://gnuwin32.sourceforge.net/packages/libgw32c.htm cheers andrew
Andrew Dunstan wrote: > > > Tom Lane wrote: > > >> While Win32 supports 64-bit files, the MinGW API does not, > >> meaning we have to build an fseeko replacement on top of the > >> Win32 API, and we have to make sure MinGW handles it. > >> > >> > > > >Wouldn't it be better to lobby the MinGW folk to fix their problem? > >Or even help them with it? I can't see the rationale for implementing > >a workaround that helps only us. > > > > > > > > > > There is a library available from the gnuwin32 project that advertises > fseeko and fseeko64. So we probably have a choice of requiring this > library or doing it ourselves. > > see http://gnuwin32.sourceforge.net/packages/libgw32c.htm OK, updated: o Add long file support for binary pg_dump output While Win32 supports 64-bit files, the MinGW API does not, meaning we have to build an fseeko replacementon top of the Win32 API, and we have to make sure MinGW handles it. Another option is to waitfor the MinGW project to fix it, or use the code from the LibGW32C project as a guide. My bet is that the license for LibGW32C might require us to use it as a guide rather than the code itself. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote: >Andrew Dunstan wrote: > > >>Tom Lane wrote: >> >> >> >>>> While Win32 supports 64-bit files, the MinGW API does not, >>>> meaning we have to build an fseeko replacement on top of the >>>> Win32 API, and we have to make sure MinGW handles it. >>>> >>>> >>>> >>>> >>>Wouldn't it be better to lobby the MinGW folk to fix their problem? >>>Or even help them with it? I can't see the rationale for implementing >>>a workaround that helps only us. >>> >>> >>> >>> >>> >>> >>There is a library available from the gnuwin32 project that advertises >>fseeko and fseeko64. So we probably have a choice of requiring this >>library or doing it ourselves. >> >>see http://gnuwin32.sourceforge.net/packages/libgw32c.htm >> >> > >OK, updated: > > o Add long file support for binary pg_dump output > > While Win32 supports 64-bit files, the MinGW API does not, > meaning we have to build an fseeko replacement on top of the > Win32 API, and we have to make sure MinGW handles it. Another > option is to wait for the MinGW project to fix it, or use the > code from the LibGW32C project as a guide. > >My bet is that the license for LibGW32C might require us to use it as a >guide rather than the code itself. > > > [after examining source] It's LGPL. But basically it just uses fsetpos() just like ours does. So maybe a little adjustment to our config so we can use ours in a 64 bit manner would be the way to go. cheers andrew