Thread: TCL with large objects on Win

<tt>Hi everybody,</tt><tt></tt><p><tt>I'm using postgres 7.1.2 with Tcl/Tk (8.3) interface on</tt><br /><tt>Windows
(thedatabase is running on Linux, only the</tt><br /><tt>clients are on Windows).</tt><tt></tt><p><tt>When using the
pg_lo_importor pg_lo_export functions,</tt><br /><tt>everything is ok under Linux (where I develop and test</tt><br
/><tt>theapplication). When I started to make a test under</tt><br /><tt>MS I recognized, that all this stuff is
broken!It won't</tt><br /><tt>import nor export binary data (while a fist test with some</tt><br /><tt>pure text and
RTFfiles worked ok). So this all is related</tt><br /><tt>to the binary stuff (CR/LF and EOF chars I
think).</tt><tt></tt><p><tt>SoI changed the scripts to use the pg_lo_create, pg_lo_open,</tt><br /><tt>pg_lo_write /
pg_lo_readand pg_lo_close together with a</tt><br /><tt>filedescriptor of the Tcl/Tk switched to binary mode:</tt><br
/><tt>   fconfigure $fd -translation binary</tt><tt></tt><p><tt>This seems to be a solution for importing the objects
(at</tt><br/><tt>least they have the correct size when I export them on the</tt><br /><tt>Linux machine), but
unfourtunatelyit doesn't work on export!</tt><br /><tt>The first problem is to find the source of this trouble -
it</tt><br/><tt>could be a Tcl/Tk problem!</tt><tt></tt><p><tt>When reading from the large object with</tt><br
/><tt>   pg_lo_read $conn $lofd buf 4096</tt><br /><tt>if returns the amount of read bytes (lets say we are at
the</tt><br/><tt>beginning of a large file, so it really reads the 4k bytes).</tt><br /><tt>When I test the length of
thestring, it will be a lot</tt><br /><tt>shorter (varies, most time it is about half the size I expected</tt><br
/><tt>itto be). Of course this leads to a corrupt file after</tt><br /><tt>complete export.</tt><tt></tt><p><tt>I also
havereduced the read size to 1 byte! In this case, the</tt><br /><tt>string length function returns often (or always
???)3!!!!</tt><tt></tt><p><tt>Because the import runs cleanly with the same data (read from</tt><br /><tt>file, written
tothe large object) I am in doubt that it really</tt><br /><tt>is a Tcl/Tk concern.</tt><tt></tt><p><tt>N.B.: While
tryingto find the reason for my trouble getting this</tt><br /><tt>all to work (blame on me: I frogot to put this all
insidea</tt><br /><tt>transaction block) I found an implemention fault in</tt><tt></tt><p><tt>   
src/interfaces/libpgtcl/pgtclCmds.c</tt><tt></tt><p><tt>Thesecond letter of the mode in Pg_lo_open() ANDs its
binary</tt><br/><tt>value to the one in the case-statement for the first letter.</tt><br /><tt>This should obviously be
anORing function like</tt><tt></tt><p><tt>    mode |= INV_READ;</tt><br /><tt>or</tt><br /><tt>    mode |=
INV_WRITE;</tt><tt></tt><p><tt>respectively.I already posted a mail about this to</tt><br
/><tt>pgsql-bugs@postgresql.orgbut maybe some of the developers</tt><br /><tt>read this here before processing the
bug-reports:-)</tt><br /><tt>Unfortunately, this has nothing to do with my problem.</tt><tt></tt><p><tt>Hope that
someonecan help me. I can't compile the windows</tt><br /><tt>DLLs (got no Microsoft compiler and don't want to use
the</tt><br/><tt>cygnus stuff). Is there a chance to compile the DLLs with</tt><br /><tt>GNU
C?</tt><tt></tt><p><tt>Bestregards</tt><br /><tt>Andreas</tt> 

Andreas Kretzer <andi@kretzer-berlin.de> writes:
> When using the pg_lo_import or pg_lo_export functions,
> everything is ok under Linux (where I develop and test
> the application). When I started to make a test under
> MS I recognized, that all this stuff is broken! It won't
> import nor export binary data (while a fist test with some
> pure text and RTF files worked ok). So this all is related
> to the binary stuff (CR/LF and EOF chars I think).

Ugh.  I'm not sure that that stuff was ever tested with Windows clients
before, and even less sure that it was tested with any recent Tcl
version.  It's very likely that you've exposed a bug.

It's possible that the problem is related to UTF8 translation that's
been imposed by recent Tcl versions.  libpgtcl's original coding
predates Tcl's adoption of UTF8, and I'm not sure that anyone has done
a careful review to look for translation issues.  So keep that in mind
while you're looking ...

> if returns the amount of read bytes (lets say we are at the
> beginning of a large file, so it really reads the 4k bytes).
> When I test the length of the string, it will be a lot
> shorter (varies, most time it is about half the size I expected
> it to be).

This definitely smells like a translation issue: is the length counted
in bytes, or characters?  Does your data contain sequences that might
be misinterpreted as UTF8 multibyte characters?

> I found an implemention fault in
>     src/interfaces/libpgtcl/pgtclCmds.c
> The second letter of the mode in Pg_lo_open() ANDs its binary
> value to the one in the case-statement for the first letter.
> This should obviously be an ORing function like
>     mode |= INV_READ;
> or
>     mode |= INV_WRITE;

Ooops.  Fixed in CVS --- thanks for the report!

> Hope that someone can help me. I can't compile the windows
> DLLs (got no Microsoft compiler and don't want to use the
> cygnus stuff). Is there a chance to compile the DLLs with
> GNU C?

No idea.  Try the folks on pgsql-cygwin...
        regards, tom lane

<tt>Tom Lane schrieb:</tt><blockquote type="CITE"><tt>Andreas Kretzer <andi@kretzer-berlin.de> writes:</tt><br
/><tt>>When using the pg_lo_import or pg_lo_export functions,</tt><br /><tt>> everything is ok under Linux (where
Idevelop and test</tt><br /><tt>> the application). When I started to make a test under</tt><br /><tt>> MS I
recognized,that all this stuff is broken! It won't</tt><br /><tt>> import nor export binary data (while a fist test
withsome</tt><br /><tt>> pure text and RTF files worked ok). So this all is related</tt><br /><tt>> to the binary
stuff(CR/LF and EOF chars I think).</tt><tt></tt><p><tt>Ugh.  I'm not sure that that stuff was ever tested with Windows
clients</tt><br/><tt>before, and even less sure that it was tested with any recent Tcl</tt><br /><tt>version.  It's
verylikely that you've exposed a bug.</tt><tt></tt><p><tt>It's possible that the problem is related to UTF8 translation
that's</tt><br/><tt>been imposed by recent Tcl versions.  libpgtcl's original coding</tt><br /><tt>predates Tcl's
adoptionof UTF8, and I'm not sure that anyone has done</tt><br /><tt>a careful review to look for translation issues. 
Sokeep that in mind</tt><br /><tt>while you're looking ...</tt><tt></tt><p><tt>> if returns the amount of read bytes
(letssay we are at the</tt><br /><tt>> beginning of a large file, so it really reads the 4k bytes).</tt><br
/><tt>>When I test the length of the string, it will be a lot</tt><br /><tt>> shorter (varies, most time it is
abouthalf the size I expected</tt><br /><tt>> it to be).</tt><tt></tt><p><tt>This definitely smells like a
translationissue: is the length counted</tt><br /><tt>in bytes, or characters?  Does your data contain sequences that
might</tt><br/><tt>be misinterpreted as UTF8 multibyte characters?</tt><br /><tt></tt> </blockquote><tt>Hmm, don't
know.What does 'string length $buf' really return in TCL?</tt><br /><tt>And YES!!! Of course there can be any character
sequencesince the data</tt><br /><tt>is straight forward binary stuff (ZIP-archives, GIF-pictures, WORD-</tt><br
/><tt>documents,etc.)</tt><tt></tt><p><tt>But anyway, thanks for your response! I will keep you informed if I
can</tt><br/><tt>find a way to build these DLLs on my own - and then I probably will fix</tt><br /><tt>this stuff and
e-mailit to you!</tt><tt></tt><p><tt>regards</tt><br /><tt>Andreas</tt><br /><tt></tt>  <br /><tt></tt>