Re: Designing a better connection pool for psycopg3 - Mailing list psycopg
| From | Vladimir Ryabtsev |
|---|---|
| Subject | Re: Designing a better connection pool for psycopg3 |
| Date | |
| Msg-id | CAMqTPqkx0rx2a79rV7UOhs1CxXZG+vG6F_qV2QWR4+9Bd+1Ocg@mail.gmail.com Whole thread Raw |
| In response to | Designing a better connection pool for psycopg3 (Daniele Varrazzo <daniele.varrazzo@gmail.com>) |
| Responses |
Re: Designing a better connection pool for psycopg3
Re: Designing a better connection pool for psycopg3 Re: Designing a better connection pool for psycopg3 |
| List | psycopg |
> Connection configuration
> What if the database is missing in action during the program lifetime? Following connection attempts may be repeated, with an exponential backoff, until dying after a few minutes of fruitless attempts
Yes, I would prefer exponential backoff, with a timeout prior to "panic" callback.
Preferably, both configurable.
> Connections usage pattern
I don't see cases where an application should bother about it,
choose whatever is better for your implementation. And yes, stack
looks reasonable.
> 'minconn', Proposed default: 4
Many services and applications do not really need many connections
but would like to benefit from automatic reconnections. So the default
is questionable, as many people will leave it unchanged.
> "get_info()"
One thing that comes into my head every time I use connection pools
is how many are really needed for my application. Usually it is adjusted
experimentally or using some performance tests. Can you collect and
provide some stats to get more insights on that? I see you want to
expose some in "get_info()", which exactly? E.g. number of times
the max limit was hit, current number of connections acquired?
> 'max_lifetime_sec'
Why is it necessary at all? Does a connection get rotten with time?
As I understand, you are planning to check a connection state in
background. As long as it is OK, what is the need to replace it?
> num_workers, default: 3
Assuming every worker does the same job, is there really the amount
of work for three workers?
"get_maintenance_task()"
This API seems to be too advanced, in what cases you see a sane use case for it?
In my opinion it is not only static vs dynamic,
> Exclusive connections?
> What to do in case of connection failure?
> A first step towards a sane behaviour could be to die early, on startup, if the connection is not working when the first pool population is done
Agree.
I have a feeling that subclassing gives more freedom for improper
implementation (resulting in problems). Also it depends on internal
API (not necessarily public), which puts more obligations on the
maintainer to keep it compatible with old releases. In this case
I like the configuring approach better
(and it does not exclude subclassing as well).
The feature itself seems useful, but what are the benefits of getting it
from the pool (e.g. checking state, reconnection, configuration)
as opposed to creating one manually?
Is such a connection counted in regard to 'minconn', 'maxconn'?
> A first step towards a sane behaviour could be to die early, on startup, if the connection is not working when the first pool population is done
Agree.
BTW, at which moment will the driver be trying to connect? At pool
initialization or on first "getconn()"?
> What if the database is missing in action during the program lifetime? Following connection attempts may be repeated, with an exponential backoff, until dying after a few minutes of fruitless attempts
Yes, I would prefer exponential backoff, with a timeout prior to "panic" callback.
Preferably, both configurable.
> Connections usage pattern
I don't see cases where an application should bother about it,
choose whatever is better for your implementation. And yes, stack
looks reasonable.
> 'minconn', Proposed default: 4
Many services and applications do not really need many connections
but would like to benefit from automatic reconnections. So the default
is questionable, as many people will leave it unchanged.
> "get_info()"
One thing that comes into my head every time I use connection pools
is how many are really needed for my application. Usually it is adjusted
experimentally or using some performance tests. Can you collect and
provide some stats to get more insights on that? I see you want to
expose some in "get_info()", which exactly? E.g. number of times
the max limit was hit, current number of connections acquired?
> 'max_lifetime_sec'
Why is it necessary at all? Does a connection get rotten with time?
As I understand, you are planning to check a connection state in
background. As long as it is OK, what is the need to replace it?
> num_workers, default: 3
Assuming every worker does the same job, is there really the amount
of work for three workers?
"get_maintenance_task()"
This API seems to be too advanced, in what cases you see a sane use case for it?
Thank you,
Vladimir
On Mon, 18 Jan 2021 at 04:04, Daniele Varrazzo <daniele.varrazzo@gmail.com> wrote:
Hello,
I've been gathering a few ideas about the connection pool I would like
to provide with psycopg3. I would be happy if you would like to take a
look and give some feedback.
https://www.psycopg.org/articles/2021/01/17/pool-design/
Thank you very much,
-- Daniele