Re: TEXT column > 1Gb - Mailing list pgsql-general

From Rob Sargent
Subject Re: TEXT column > 1Gb
Date
Msg-id d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com
Whole thread Raw
In response to Re: TEXT column > 1Gb  (Joe Carlson <jwcarlson@lbl.gov>)
List pgsql-general
On 4/12/23 08:59, Joe Carlson wrote:
> I’m curious what you learned. I’ve been tripping over the buffer 
> allocation issue when either splitting input text into chunks or 
> aggregating chunks in selects. I’ve decided that I need to move this 
> to client side.
>
> The use case is genomics. Extracting substrings is common. So going to 
> chunked storage makes sense.
>
> And we have a menagerie of code to deal with. Legacy Perl for loading. 
> Clients in Perl, Java, node, python. A whole zoo.
>
> Thanks

My first stab was a java implementation based Selector class which 
worked ok using a fixed buffer size but finding the end of last 
transmission was hokey.  I move to an embedded Tomcat implementation and 
tomcat does the heavy lifting PERFECTLY.  Easily slipped in to both 
client and backend.

rjs




pgsql-general by date:

Previous
From: Adrian Klaver
Date:
Subject: Re: FW: Error!
Next
From: Mark Dilger
Date:
Subject: Re: TEXT column > 1Gb