Re: fast read of binary data - Mailing list pgsql-performance

From Albe Laurenz
Subject Re: fast read of binary data
Date
Msg-id D960CB61B694CF459DCFB4B0128514C208AF0471@exadv11.host.magwien.gv.at
Whole thread Raw
In response to fast read of binary data  (Eildert Groeneveld <eildert.groeneveld@fli.bund.de>)
Responses Re: fast read of binary data  (Eildert Groeneveld <eildert.groeneveld@fli.bund.de>)
List pgsql-performance
Eildert Groeneveld wrote:
> I am currently implementing using a compressed binary storage scheme
> genotyping data. These are basically vectors of binary data which may be
> megabytes in size.
> 
> Our current implementation uses the data type bit varying.
> 
> What we want to do is very simple: we want to retrieve such records from
> the database and transfer it unaltered to the client which will do
> something (uncompressing) with it. As massive amounts of data are to be
> moved, speed is of great importance, precluding any to and fro
> conversions.
> 
> Our current implementation uses Perl DBI; we can retrieve the data ok,
> but apparently there is some converting going on.
> 
> Further, we would like to use ODBC from Fortran90 (wrapping the
> C-library)  for such transfers. However, all sorts funny things happen
> here which look like conversion issues.
> 
> In old fashioned network database some decade ago (in pre SQL times)
> this was no problem. Maybe there is someone here who knows the PG
> internals sufficiently well to give advice on how big blocks of memory
> (i.e. bit varying records) can between transferred UNALTERED between
> backend and clients.

Using the C API you can specify binary mode for your data, which
meand that they won't be converted.

I don't think you will be able to use this with DBI or ODBC,
but maybe binary corsors can help
(http://www.postgresql.org/docs/current/static/sql-declare.html),
but I don't know if DBI or ODBC handles them well.

If you can avoid DBI or ODBC, that would be best.

Yours,
Laurenz Albe

pgsql-performance by date:

Previous
From: Eildert Groeneveld
Date:
Subject: fast read of binary data
Next
From: Arjen van der Meijden
Date:
Subject: Re: fast read of binary data