Thread: [ODBC] ODBC large binary data support

Hi,
On 2017/01/04 23:34, myaddress@gmx-topmail.de wrote:

Hello,

 

Recently I successfully stored a 3GB BLOB in PostgreSQL V9.6.1 via psql-odbc V09.05.0400, but I was not able to read it afterwards. I got the error message "Error reading from large object." for the very first SQLGetData call. I debugged it and found out that psql-odbc uses "lo_lseek" and "lo_tell" in "lobj.c" which limit the amount of data to 2GB (int32 usage). In "lobj.c" there is even the comment /* XXX: Should we use lo_lseek64? */ and /* XXX: Should we use lo_tell64? */.

I temporarily changed the code to use lo_lseek64 and lo_tell64 and I could read the 3GB BLOB.

So, could you please change the official psql-odbc to use lo_lseek64 and lo_tell64?

 

I found a discussion about that here:

https://www.postgresql.org/message-id/CAB7nPqSc7M_wJYjZEfQ7QyBg1mbo3_Dhg0Jeiqx7HpO+46HFTA@mail.gmail.com

 

It tells that the 64-bit functions are not supported prior to PostgreSQL V9.3. My suggestion would be either dropping the support for old versions or a dynamic switch between the 64-bit and the 32-bit functions based on the server version.

I hope that you support that.

 

Dear Hiroshi Inoue,

 

> On what OS are you working?

 

Windows 10 and Windows 7.
 

> Could you test my code?

 

Yes, I can do that.

I'm using the 32-bit as well as the 64-bit variant of ODBC. This means I can test it with both variants.

> Hopefully I would support that.

 

Thank you very much!

Dear Hiroshi Inoue,

 

Just to avoid any confusion: I used the wrong sending e-mail address for my last answer. But it's the same person :-)

 

It's working for the 64-bit version. Unfortunately, it's not working correctly for the 32-bit version. The problem is the last parameter of "SQLGetData". For the 32-bit version this is a pointer to a 32-bit signed integer (SQLLEN). This means, the integer overflows for large data >2 GB. I suggest to saturate the value for the 32-bit version and return 0x7fffffff in case the real value is bigger than that. Most applications just check if it's bigger than the provided buffer size and for that returning 0x7fffffff works fine. What do you think about it?

Dear Hiroshi Inoue,

 

The new update fixed my issues. It's also working with the 32-bit version now. The Microsoft documentation concerning SQL_NO_TOTAL is a little bit strange in my eyes.

So, I don't know what the "correct" behavior is for the 32-bit version and data >2GB. My feeling teels me that SQL_NO_TOTAL is "correcter" than 0x7fffffff. So I'd say the current version is good.

When will you officially release it?

7. Places the length of the data in *StrLen_or_IndPtr. If StrLen_or_IndPtr was a null pointer, SQLGetData does not return the length.     For character or binary data, this is the length of the data after conversion and before truncation due to BufferLength. If the driver
    cannot determine the length of the data after conversion, as is sometimes the case with long data, it returns
    SQL_SUCCESS_WITH_INFO and sets the length to SQL_NO_TOTAL. (The last call to SQLGetData must always return the length of the
    data, not zero or SQL_NO_TOTAL.)

The test driversI return SQL_NO_TOTAL while the length of the rest of the data is outside the range of SQLLEN.

regards,
Hiroshi Inoue