Re: [GENERAL] Urgent: 10K or more connections - Mailing list pgsql-hackers
From | Sean Chittenden |
---|---|
Subject | Re: [GENERAL] Urgent: 10K or more connections |
Date | |
Msg-id | 20030720105326.GM24507@perrin.int.nxad.com Whole thread Raw |
List | pgsql-hackers |
[ moved to -hackers ] > > No where, everything remains as is. I actually think you'll > > appreciate the simplicity of this once I'm done explaining how I'm > > going about this. > > I don't think you heard a word I said :-( Heh, I'm thinking the same thing right now. :-) > You appear to be adding a whole lot of platform dependency (passing > FDs around other than by fork() is not portable) Apache's been doing this for years now and does it unconditionally (in what I'm working on, it's pretty easily #ifdef'ed out), which includes the following OSes: Linux, SunOS, UnixWare, Darwin/Mac OS, FreeBSD, Solaris, AIX, OpenStep/Mach, OpenBSD, IRIX, SCO, DYNIX/ptx, NetBSD, HPUX, ReliantUNIX, BSDI, Digital Unix, and DGUX. The only OS that doesn't support this is Ultrix. Passing FD's seems to be pretty portable to me and it's far from a new concept in the UNIX world. Win32 can't do this, which is why all of this code is going to be #ifdef'ed appropriately. > in return for loss of functionality (applications can't rely on > session state anymore) They can rely on it because an FDs is only passed back to the parent when the client calls PQfinish(). I think you're under the impression I'm implementing that half baked idea about having this happen along transaction borders (which would work for my current situation since most all of my apps are aligned inside of transactions and don't depend on a particular backend hanging around, but it's far from the correct way of doing things). What I described in my last email is a bit different and lets apps continue to depend on the backends being there for as long as the lib keeps the connection "open". PQfinish() is more of a, "make this connection idle until I need it again" call. > and highly dubious performance gain (just because a backend has > started does not mean it's built up a reasonable working set of > cache entries; I don't think it's that dubious. A freshly started backend doesn't have much in the way of cached data with the way that things exist currently. If backends are sitting idle in a pool and as much code as possible is transplanted before a connection is authenticated, obviously the amount of time required to establish a connection goes down. It's not _tons_ of code, but, from what I can tell, these steps in BackendFork() could be moved to the new BackendInit(): 1) ClosePostmasterPorts() 2) pq_init() 3) getnameinfo() 4) pqsignal() 5) srandom() 6) MemoryContextSwitchTo()/MemoryContextDelete() 7) pqsignal(), again, different handlers though 8) AttachSharedMemoryAndSemaphores() 9) AllocSetContextCreate() 10) BaseInit() which leaves the following code to be executed when a connection does come in: 1) ClientAuthentication() 2) SetProcessingMode() 3) getopt() 4) A bit of logging and simple if statements for logging/stand along backend startup 5) InitPostgres() 6) sigsetjmp() Of which, InitPostgres() and sigsetjmp() are the most expensive and it looks like there are a few bits in InitPostgres() that could be moved to BaseInit() without much trouble. > you can't expect that firing off a new backend for every transaction > is going to be anything but a huge performance loss, even if you > assume its visible session state is exactly what the application > needs). *nods* That's not what I'm advocating doing though. Interestingly enough, however, this is exactly what SQLRelay does and there are a number of reports from SQLRelay's users suggesting it's not all bad... though it does pretty radically break some of the assumptions developers should make about their backend. Remember though, what I'm advocating for is to pass FDs back to the parent when a client calls PQfinish(), not in between transactions. > Also it sounds to me like the postmaster will now become a > performance bottleneck, since it will need to be involved in every > transaction start. Well, I'm pretty sure you're under the impression I'm chasing a different goal than the one I'm working toward. -sc -- Sean Chittenden
pgsql-hackers by date: