Re: libpq: Process buffered SSL read bytes to support records >8kB on async API - Mailing list pgsql-hackers

From Merlin Moncure
Subject Re: libpq: Process buffered SSL read bytes to support records >8kB on async API
Date
Msg-id CAHyXU0w1qUzxmXPa30FBNFu+wjLOCvU0N6ezuE0XK+eWWEugTQ@mail.gmail.com
Whole thread Raw
List pgsql-hackers
On Sun, Sep 8, 2024 at 2:08 PM Lars Kanis <lars@greiz-reinsdorf.de> wrote:
To fix this issue the attached patch calls pqReadData() repeatedly in
PQconsumeInput() until there is no buffered SSL data left to be read.
Another solution could be to process buffered SSL read bytes in
PQisBusy() instead of PQconsumeInput() .
 
I've definitely seen this problem in AWS RDS, or something related to this.   I'm using dblink to set up lots of connections and using dblink_is_busy (which calls PQconsumeInput).  If there is lots of data being returned, say, with large results or log information, the server would stall for unknown reasons (sometimes for several seconds or more) while peeling off the 8kb fragments.  Point being, PQconsumeInput() not draining the buffers can lead to major performance problems that are hard to troubleshoot.  I would argue this a bug, really.

merlin

pgsql-hackers by date:

Previous
From: Masahiko Sawada
Date:
Subject: Re: Inconsistent LSN format in pg_waldump output
Next
From: "Hayato Kuroda (Fujitsu)"
Date:
Subject: RE: A assert failure when initdb with track_commit_timestamp=on