Thread: connection file descriptors created with identical number after process fork on mac
connection file descriptors created with identical number after process fork on mac
From
Chris Withers
Date:
Hi All,
I'm writing some multi-process code in Python and trying to make sure I
open a new connection for each process. Here's the really cut down code:
"""
import os, time
import psycopg2
from multiprocessing import Pool
def init():
conn = psycopg2.connect("dbname=...host=localhost")
print os.getpid(), ' child fd:', conn.fileno()
if __name__=='__main__':
pool = Pool(initializer=init)
time.sleep(30)
"""
What's really surpising to me is the output on a mac:
$ python psycopg2_multiprocess.py
44276 child fd: 13
44277 child fd: 13
44278 child fd: 13
44279 child fd: 13
The getpid() output indicates that the connec() call is being made
inside a different process each time, yet the connection appears to
still be using the same fd.
conn.file() is basically (long int)PQsocket(self->pgconn);:
https://github.com/psycopg/psycopg2/blob/master/psycopg/connection_type.c#L898
Is there something I'm missing about file descriptors on Macs or is
something bad happening here?
Chris
Re: connection file descriptors created with identical number after process fork on mac
From
Tom Lane
Date:
Chris Withers <chris@simplistix.co.uk> writes:
> I'm writing some multi-process code in Python and trying to make sure I
> open a new connection for each process. Here's the really cut down code:
> ...
> What's really surpising to me is the output on a mac:
> $ python psycopg2_multiprocess.py
> 44276 child fd: 13
> 44277 child fd: 13
> 44278 child fd: 13
> 44279 child fd: 13
> The getpid() output indicates that the connec() call is being made
> inside a different process each time, yet the connection appears to
> still be using the same fd.
FD numbers are process-local in all flavors of Unix. The above only
proves that all of these processes had FDs 0..12 open already, which
doesn't seem terribly surprising.
regards, tom lane
Re: connection file descriptors created with identical number after process fork on mac
From
Chris Withers
Date:
On 04/08/2016 00:20, Tom Lane wrote: > Chris Withers <chris@simplistix.co.uk> writes: >> I'm writing some multi-process code in Python and trying to make sure I >> open a new connection for each process. Here's the really cut down code: >> ... >> What's really surpising to me is the output on a mac: > >> $ python psycopg2_multiprocess.py >> 44276 child fd: 13 >> 44277 child fd: 13 >> 44278 child fd: 13 >> 44279 child fd: 13 > >> The getpid() output indicates that the connec() call is being made >> inside a different process each time, yet the connection appears to >> still be using the same fd. > > FD numbers are process-local in all flavors of Unix. The above only > proves that all of these processes had FDs 0..12 open already, which > doesn't seem terribly surprising. Thanks, that's certainly good news! How can I convince myself, from the client side, that I really have got a new connection and not somehow ended up with one that been passed on as part of the fork? cheers, Chris
Re: connection file descriptors created with identical number after process fork on mac
From
John R Pierce
Date:
On 8/3/2016 4:25 PM, Chris Withers wrote:
>
> How can I convince myself, from the client side, that I really have
> got a new connection and not somehow ended up with one that been
> passed on as part of the fork?
$ psql -tc "select pg_backend_pid();"
18635
$ psql -tc "select pg_backend_pid();"
18665
$ psql -tc "select pg_backend_pid();"
18727
now, operating system process ID's do get recycled eventually but not
very fast.
--
john r pierce, recycling bits in santa cruz