Thread: [HACKERS] PATCH: pgbench - option to build using ppoll() for larger connectioncounts
[HACKERS] PATCH: pgbench - option to build using ppoll() for larger connectioncounts
From
"Rady, Doug"
Date:
This patch enables building pgbench to use ppoll() instead of select()
to allow for more than (FD_SETSIZE - 10) connections. As implemented,
when using ppoll(), the only connection limitation is system resources.
The patch has been implemented to introduce a minimal of #ifdef/#ifndef
clutter in the code.
Two patches attached.
One based on REL9_6_STABLE.
One based on 'master' which can also apply to REL_10_STABLE.
doug
--
Doug Rady
Amazon Aurora PostgreSQL
radydoug@amazon.com
Attachment
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Andres Freund
Date:
On 2017-09-25 18:01:40 +0000, Rady, Doug wrote: > This patch enables building pgbench to use ppoll() instead of select() > to allow for more than (FD_SETSIZE - 10) connections. As implemented, > when using ppoll(), the only connection limitation is system resources. Hm, is there any need of using ppoll over poll? IIRC it's a good bit more common and there's, also iirc, a number of platforms with buggy ppoll implementations. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Fabien COELHO
Date:
Hello Again, > Two patches attached. > One based on REL9_6_STABLE. I'd be surprise that there would be a backport unless there is a bug, so this one might not be useful. > One based on 'master' which can also apply to REL_10_STABLE. Could you add your patches to the next CF? -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
"Rady, Doug"
Date:
On 9/25/17, 11:07, "Andres Freund" <andres@anarazel.de> wrote: On 2017-09-25 18:01:40 +0000, Rady, Doug wrote: > This patch enables building pgbench to use ppoll() instead of select() > to allow for more than (FD_SETSIZE - 10) connections. As implemented, > when using ppoll(), the only connectionlimitation is system resources. Hm, is there any need of using ppoll over poll? IIRC it's a good bit morecommon and there's, also iirc, a number of platforms with buggy ppoll implementations. Greetings, AndresFreund I used ppoll() as it can support the microseconds duration for \SLEEP meta command. thanks! doug -- Doug Rady Amazon Aurora PostgreSQL radydoug@amazon.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Fabien COELHO
Date:
> This patch enables building pgbench to use ppoll() instead of select()
> to allow for more than (FD_SETSIZE - 10) connections. As implemented,
> when using ppoll(), the only connection limitation is system resources.
>
> One based on 'master' which can also apply to REL_10_STABLE.
/home/fabien/pgbench-ppoll.patch:137: trailing whitespace. #define PFD_THREAD_INIT(t,s,n) { do ... error: patch
failed:configure:13024 error: configure: patch does not apply error: patch failed: configure.in:1430 error:
configure.in:patch does not apply error: patch failed: src/bin/pgbench/pgbench.c:4588 error: src/bin/pgbench/pgbench.c:
patchdoes not apply
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
"Rady, Doug"
Date:
Fixed the trailing garbage … sigh.
Fixed the init of unused pollfds.
Re-based for both 10 & master.
--
Thank you!
doug
On 10/3/17, 23:21, "Fabien COELHO" <coelho@cri.ensmp.fr> wrote:
> This patch enables building pgbench to use ppoll() instead of select()
> to allow for more than (FD_SETSIZE - 10) connections. As implemented,
> when using ppoll(), the only connection limitation is system resources.
>
> One based on 'master' which can also apply to REL_10_STABLE.
/home/fabien/pgbench-ppoll.patch:137: trailing whitespace.
#define PFD_THREAD_INIT(t,s,n) { do ...
error: patch failed: configure:13024
error: configure: patch does not apply
error: patch failed: configure.in:1430
error: configure.in: patch does not apply
error: patch failed: src/bin/pgbench/pgbench.c:4588
error: src/bin/pgbench/pgbench.c: patch does not apply
--
Fabien.
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Robert Haas
Date:
On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote: > This patch enables building pgbench to use ppoll() instead of select() > > to allow for more than (FD_SETSIZE - 10) connections. As implemented, > > when using ppoll(), the only connection limitation is system resources. So what's an example of something that fails without this patch but works with the patch? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
"Rady, Doug"
Date:
Without this patch, one is limited to '(FD_SETSIZE - 10)’ number of connections. Example of something that fails without this patch but works with the patch: Without the patch: $ pgbench -j 3000 -c 1500 invalid number of clients: "1500" With the patch: $ pgbench -j 3000 -c 1500 starting vacuum...end. transaction type: <builtin: TPC-B (sort of)> scaling factor: 2000 query mode: simple number of clients: 1500 number of threads: 1500 number of transactions per client: 10 number of transactions actually processed: 15000/15000 latency average = 631.730 ms tps = 2374.430587 (including connections establishing) tps = 4206.524986 (excluding connections establishing) -- doug On 10/26/17, 04:46, "Robert Haas" <robertmhaas@gmail.com> wrote: On Mon, Sep 25, 2017 at 8:01 PM, Rady, Doug <radydoug@amazon.com> wrote: > This patch enables building pgbench to useppoll() instead of select() > > to allow for more than (FD_SETSIZE - 10) connections. As implemented, > > whenusing ppoll(), the only connection limitation is system resources. So what's an example of something that failswithout this patch but works with the patch? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Fabien COELHO
Date:
Hello, Could you rebase the v11 patch? -- Fabien. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Michael Paquier
Date:
On Fri, Nov 3, 2017 at 2:29 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote: > Could you rebase the v11 patch? This patch has been waiting for a rebase for more than three weeks as of today, I am marking it as returned with feedback. It would be a good idea to reply to Robert's input in https://www.postgresql.org/message-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com. -- Michael
Re: [HACKERS] PATCH: pgbench - option to build using ppoll() forlarger connection counts
From
Fabien COELHO
Date:
Hello Michaël, >> Could you rebase the v11 patch? > > This patch has been waiting for a rebase for more than three weeks as > of today, I am marking it as returned with feedback. It would be a > good idea to reply to Robert's input in > https://www.postgresql.org/message-id/CA+TgmoYybNv-DdhVPMxLB2nx2SqeNJirtWHmdEAZUCGoTB2VBg@mail.gmail.com. ISTM that this was done: If -c is high enough, pgbench fails without the patch. -- Fabien.