Thread: BUG #13462: Impossible to use COPY FORMAT BINARY in chunks through libpq
BUG #13462: Impossible to use COPY FORMAT BINARY in chunks through libpq
From
socketpair@gmail.com
Date:
The following bug has been logged on the website: Bug reference: 13462 Logged by: Korenberg Mark Email address: socketpair@gmail.com PostgreSQL version: 9.3.8 Operating system: Linux Description: If I use PQputCopyData() to pass parts of the binary protocol (as documented in http://www.postgresql.org/docs/9.3/static/sql-copy.html ), postgres binary socket data gets corrupted (i.e. protocol is broken). It is unexpected that libpq inserts \d\0\0\0 on every PQputCopyData() call even in BINARY COPY mode. So, it is impossible to write a program that generate tuples in binary form dynamically without merging them into one big buffer. Part of program: ==================================== static const struct { char magic[11]; int32_t zero1; int32_t zero2; } __attribute__ ((packed)) bin_header = {"PGCOPY\n\377\r\n",0,0}; if (PQputCopyData (conn, (const char *) &bin_header, sizeof (bin_header)) != 1) { fprintf (stderr, "PQputCopyData failed\n"); return 4; } struct { int16_t fields; int32_t ip1len; uint32_t ip1; int32_t ip2len; uint32_t ip2; } __attribute__ ((packed)) binary_line = { 2, 4, 0x01020304, 4, 0x05060708 }; for (i = 0; i < repeats; i++) { if (PQputCopyData (conn, (const char *) &binary_line, sizeof (binary_line)) != 1) { fprintf (stderr, "PQputCopyData failed\n"); return 4; } } static const int16_t copy_end = -1; if (PQputCopyData (conn, (const char *) ©_end, sizeof (copy_end)) != 1) { fprintf (stderr, "PQputCopyData failed\n"); return 4; } res = PQgetResult (conn); if (PQresultStatus (res) != PGRES_COMMAND_OK) { fprintf (stderr, "COPY failed: %s\n", PQerrorMessage (conn)); PQclear (res); return 3; } PQclear (res); =============================================== will generate: "d\0\0\0\27PGCOPY\n\377\r\n\0\0\0\0\0\0\0\0\0d\0\0\0\26\2\0\4\0\0\0\4\3\2\1\4\0\0\0\10\7\6\5d\0\0\0\6\377\377X\0\0\0\4" but actually should generate (not sure about tail) "d\0\0\0\27PGCOPY\n\377\r\n\0\0\0\0\0\0\0\0\0\2\0\4\0\0\0\4\3\2\1\4\0\0\0\10\7\6\5\377\377X\0\0\0\4"
socketpair@gmail.com writes: > If I use PQputCopyData() to pass parts of the binary protocol (as documented > in http://www.postgresql.org/docs/9.3/static/sql-copy.html ), postgres > binary socket data gets corrupted (i.e. protocol is broken). I think your test program is buggy; specifically, it doesn't appear to have any provision for dealing with data endianness, which means it will not work on a little-endian machine. Per the COPY binary format spec: Headers and data are in network byte order. regards, tom lane
Re: BUG #13462: Impossible to use COPY FORMAT BINARY in chunks through libpq
From
Марк Коренберг
Date:
Thanks, but still not works.... How to debug ? strace: sendto(3, "d\0\0\0\27PGCOPY\n\377\r\n\0\0\0\0\0\0\0\0\0d\0\0\0\26\0\2\0\0\0\4\4\3\2\1\0\0\0\4\10\7\6\5d\0\0\0\6\377\377X\0\0\0\4", 59, MSG_NOSIGNAL, NULL, 0) = 59 2015-06-23 4:08 GMT+05:00 Tom Lane <tgl@sss.pgh.pa.us>: > socketpair@gmail.com writes: >> If I use PQputCopyData() to pass parts of the binary protocol (as documented >> in http://www.postgresql.org/docs/9.3/static/sql-copy.html ), postgres >> binary socket data gets corrupted (i.e. protocol is broken). > > I think your test program is buggy; specifically, it doesn't appear to > have any provision for dealing with data endianness, which means it > will not work on a little-endian machine. Per the COPY binary format > spec: > > Headers and data are in network byte order. > > > regards, tom lane -- Segmentation fault
Re: BUG #13462: Impossible to use COPY FORMAT BINARY in chunks through libpq
From
Марк Коренберг
Date:
Where I can see example how to generate COPY FORMAT BINARY ? Seems, I'm doing something wrong, but cannot figure out what exactly. According to strace, library act as follow: sendto ("COPY.....FROM STDIN (FORMAT binary)") recv (...) => receive some response (that copy starts succesfully) next, debugging and logging of PQputCopyData calls logging of calling PQgetResult, that reports IMMEDIATELLY, that COPY failed. and AFTER that, library sends "\d\0\0\0\27PGCOPY\n\377\r\n\0\0\....." WTF? If my example will work, I want to contribute that example to documentation (as canonical example of that stuff), since I cannot find working example of generating binary copy data. 2015-06-23 4:08 GMT+05:00 Tom Lane <tgl@sss.pgh.pa.us>: > socketpair@gmail.com writes: >> If I use PQputCopyData() to pass parts of the binary protocol (as documented >> in http://www.postgresql.org/docs/9.3/static/sql-copy.html ), postgres >> binary socket data gets corrupted (i.e. protocol is broken). > > I think your test program is buggy; specifically, it doesn't appear to > have any provision for dealing with data endianness, which means it > will not work on a little-endian machine. Per the COPY binary format > spec: > > Headers and data are in network byte order. > > > regards, tom lane -- Segmentation fault