Re: [HACKERS] [bug-fix] Cannot select big bytea values (~600MB) - Mailing list pgsql-hackers

From Tom Lane
Subject Re: [HACKERS] [bug-fix] Cannot select big bytea values (~600MB)
Date
Msg-id 26590.1518799293@sss.pgh.pa.us
Whole thread Raw
In response to Re: [HACKERS] [bug-fix] Cannot select big bytea values (~600MB)  (Andres Freund <andres@anarazel.de>)
List pgsql-hackers
Andres Freund <andres@anarazel.de> writes:
> On 2018-02-16 09:58:29 -0500, Tom Lane wrote:
>> Anna Akenteva <a.akenteva@postgrespro.ru> writes:
>>> [ widen StringInfoData max length to size_t ]

>> I find this scary as heck.  Have you spent any time looking at the
>> side effects?  There are probably hundreds of places that expect that
>> stringinfos won't get larger than 1GB.

> FWIW, I think we're going to have to bite that bullet sooner rather than
> later. I do agree it's not going to fix this issue for real, and that
> we're not going to backpatch it.

I'm not necessarily saying we shouldn't consider widening this.
I'm just saying it's going to take a good deal of cross-checking for
consequences, in particular that nothing is at risk of integer overflow
if it's presented with a very long StringInfo.

One way to limit the side effects would be to have StringInfos default to
only allowing 1GB of content as before, and you have to do something extra
at creation time to let one get bigger.

There's still the problem that the wire protocol will limit us to 2GB
(or maybe 4GB if you want to be brave^Wfoolhardy and assume clients think
the width fields are unsigned).  I can't get hugely excited about moving
the goalposts only from 1GB to 2GB ...

            regards, tom lane


pgsql-hackers by date:

Previous
From: Tom Lane
Date:
Subject: Re: master check fails on Windows Server 2008
Next
From: Alvaro Herrera
Date:
Subject: Re: [HACKERS] [bug-fix] Cannot select big bytea values (~600MB)