Thread: Using Threads?
Hello all, I am new to postgreSQL. When I read the documents, I find out the Postmaster daemon actual spawns a new backend server process to serve a new client request. Why not use threads instead? Is that just for a historical reason, or some performance/implementation concern? Thank you very much. Junfeng
> I am new to postgreSQL. When I read the documents, I find out the Postmaster
> daemon actual spawns a new backend server process to serve a new client
> request. Why not use threads instead? Is that just for a historical reason,
> or some performance/implementation concern?
Both. Not all systems supported by PostgreSQL have a standards-compliant
threading implementation (even more true for the systems PostgreSQL has
supported over the years).
But there are performance and reliability considerations too. A
thread-only server is likely more brittle than a process-per-client
implementation, since all threads share the same address space.
Corruption in one server might more easily propagate to other servers.
The time to start a backend is quite often small compared to the time
required for a complete session, so imho the differences in absolute
speed are not generally significant.
- Thomas
On Mon, 27 Nov 2000, Junfeng Zhang wrote:
> Hello all,
>
> I am new to postgreSQL. When I read the documents, I find out the Postmaster
> daemon actual spawns a new backend server process to serve a new client
> request. Why not use threads instead? Is that just for a historical reason,
> or some performance/implementation concern?
It's a little a historical reason, but not only. The PostgreSQL allowsto use user defined modules (functions), it means
thatbad module orbug in core code crash one backend only, but postmaster run still. In the thread model crash all
runningbackend. Big differntion is in the lock method too.
Karel
I maybe wrong but I think that PGSQL is not threaded mostly due to historical reasons. It looks to me like the source has developed over time where much of the source is not reentrant with many global variables throughout. In addition, the parser is generated by flex which can be made to generate reentrant code but is still not thread safe b/c global variables are used. That being said, I experimented with the 7.0.2 source and came up with a multithreaded backend for PGSQL which uses Solaris Threads. It seems to work, but I drifted very far from the original source. I had to hack flex to generate threadsafe code as well. I use it as a linked library with my own fe<->be protocol. This ended up being much much more than I bargained for and looking back would probably not have tried had I known any better. Myron Scott On Mon, 27 Nov 2000, Junfeng Zhang wrote: > Hello all, > > I am new to postgreSQL. When I read the documents, I find out the Postmaster > daemon actual spawns a new backend server process to serve a new client > request. Why not use threads instead? Is that just for a historical reason, > or some performance/implementation concern? > > Thank you very much. > Junfeng >
On Mon, 27 Nov 2000, Junfeng Zhang wrote: > Hello all, > > I am new to postgreSQL. When I read the documents, I find out the > Postmaster daemon actual spawns a new backend server process to serve > a new client request. Why not use threads instead? Is that just for a > historical reason, or some performance/implementation concern? Several reasons, 'historical' probably being the strongest right now ... since PostgreSQL was never designed for threading, its about as 'un-thread-safe' as they come, and cleaning that up will/would be a complete nightmare (should eventually be done, mind you) ... The other is stability ... right now, if one backend drops away, for whatever reason, it doesn't take down the whole system ... if you ran things as one process, and that one process died, you just lost your whole system ...
Myron - Putting aside the fork/threads discussion for a moment (the reasons, both historical and other, such as inter-backend protection, are well covered in the archives), the work you did sounds like an interesting experiment in code redesign. Would you be willing to release the hacked code somewhere for others to learn from? Hacking flex to generate thread-safe code is of itself interesting, and the question about PG and threads comes up so often, that an example of why it's not a simple task would be useful. Ross On Mon, Dec 04, 2000 at 12:20:20AM -0800, Myron Scott wrote: > I maybe wrong but I think that PGSQL is not threaded mostly due to > historical reasons. It looks to me like the source has developed over > time where much of the source is not reentrant with many global variables > throughout. In addition, the parser is generated by flex which > can be made to generate reentrant code but is still not thread safe b/c > global variables are used. > > That being said, I experimented with the 7.0.2 source and came up with a > multithreaded backend for PGSQL which uses Solaris Threads. It seems to > work, but I drifted very far from the original source. I > had to hack flex to generate threadsafe code as well. I use it as a > linked library with my own fe<->be protocol. This ended up being much much > more than I bargained for and looking back would probably not have tried > had I known any better. > > > Myron Scott >
if we were to do this in steps, I beliee that one of the major problems
irght now is that we have global variables up the wazoo ... my
'thread-awareness' is limited, as I've yet to use them, so excuse my
ignorance ... if we got patches that cleaned up the code in stages, moving
towards a cleaner code base, then we could get it into the main source
tree ... ?
On Mon, 4 Dec 2000, Ross J. Reedstrom wrote:
> Myron -
> Putting aside the fork/threads discussion for a moment (the reasons,
> both historical and other, such as inter-backend protection, are well
> covered in the archives), the work you did sounds like an interesting
> experiment in code redesign. Would you be willing to release the hacked
> code somewhere for others to learn from? Hacking flex to generate
> thread-safe code is of itself interesting, and the question about PG and
> threads comes up so often, that an example of why it's not a simple task
> would be useful.
>
> Ross
>
> On Mon, Dec 04, 2000 at 12:20:20AM -0800, Myron Scott wrote:
> > I maybe wrong but I think that PGSQL is not threaded mostly due to
> > historical reasons. It looks to me like the source has developed over
> > time where much of the source is not reentrant with many global variables
> > throughout. In addition, the parser is generated by flex which
> > can be made to generate reentrant code but is still not thread safe b/c
> > global variables are used.
> >
> > That being said, I experimented with the 7.0.2 source and came up with a
> > multithreaded backend for PGSQL which uses Solaris Threads. It seems to
> > work, but I drifted very far from the original source. I
> > had to hack flex to generate threadsafe code as well. I use it as a
> > linked library with my own fe<->be protocol. This ended up being much much
> > more than I bargained for and looking back would probably not have tried
> > had I known any better.
> >
> >
> > Myron Scott
> >
>
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
The Hermit Hacker <scrappy@hub.org> writes:
>> Why not use threads instead? Is that just for a
>> historical reason, or some performance/implementation concern?
> Several reasons, 'historical' probably being the strongest right now
> ... since PostgreSQL was never designed for threading, its about as
> 'un-thread-safe' as they come, and cleaning that up will/would be a
> complete nightmare (should eventually be done, mind you) ...
> The other is stability ... right now, if one backend drops away, for
> whatever reason, it doesn't take down the whole system ... if you ran
> things as one process, and that one process died, you just lost your whole
> system ...
Portability is another big reason --- using threads would create lots
of portability headaches for platforms that had no threads or an
incompatible threads library. (Not to mention buggy threads libraries,
not-quite-thread-safe libc routines, yadda yadda.)
The amount of work required looks far out of proportion to the payoff...
regards, tom lane
On Mon, Nov 27, 2000 at 11:42:24PM -0600, Junfeng Zhang wrote: > I am new to postgreSQL. When I read the documents, I find out the Postmaster > daemon actual spawns a new backend server process to serve a new client > request. Why not use threads instead? Is that just for a historical reason, > or some performance/implementation concern? Once all the questions regarding "why not" have been answered, it would be good to also ask "why use threads?" Do they simplify the code? Do they offer significant performance or efficiency gains? What do they give, other than being buzzword compliant? -- Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/
All the major operating systems should have POSIX threads implemented. Actually this can be configurable--multithreads or one thread. Thread-only server is unsafe, I agree. Maybe the following model can be a little better. Several servers, each is multi-threaded. Every server can support a maximum number of requests simultaneously. If anything bad happends, it is limited to that server. The cons side of processes model is not the startup time. It is about kernel resource and context-switch cost. Processes consume much more kernel resource than threads, and have a much higher cost for context switch. The scalability of threads model is much better than that of processes model. -Junfeng On Mon, 4 Dec 2000, Thomas Lockhart wrote: > > I am new to postgreSQL. When I read the documents, I find out the Postmaster > > daemon actual spawns a new backend server process to serve a new client > > request. Why not use threads instead? Is that just for a historical reason, > > or some performance/implementation concern? > > Both. Not all systems supported by PostgreSQL have a standards-compliant > threading implementation (even more true for the systems PostgreSQL has > supported over the years). > > But there are performance and reliability considerations too. A > thread-only server is likely more brittle than a process-per-client > implementation, since all threads share the same address space. > Corruption in one server might more easily propagate to other servers. > > The time to start a backend is quite often small compared to the time > required for a complete session, so imho the differences in absolute > speed are not generally significant. > > - Thomas >
On Mon, Dec 04, 2000 at 02:28:10PM -0600, Bruce Guenter wrote: > On Mon, Nov 27, 2000 at 11:42:24PM -0600, Junfeng Zhang wrote: > > I am new to postgreSQL. When I read the documents, I find out the Postmaster > > daemon actual spawns a new backend server process to serve a new client > > request. Why not use threads instead? Is that just for a historical reason, > > or some performance/implementation concern? > > Once all the questions regarding "why not" have been answered, it would > be good to also ask "why use threads?" Do they simplify the code? Do > they offer significant performance or efficiency gains? What do they > give, other than being buzzword compliant? Typically (on a well-written OS, at least), the spawning of a thread is much cheaper then the creation of a new process (via fork()). Also, since everything in a group of threads (I'll call 'em a team) shares the same address space, there can be some memory overhead savings. -- Adam Haberlach |"California's the big burrito, Texas is the big adam@newsnipple.com | taco ... and following that theme, Florida is http://www.newsnipple.com| the big tamale ... and the only tamale that '88 EX500 | counts any more." -- Dan Rather
Adam Haberlach writes: > Typically (on a well-written OS, at least), the spawning of a thread > is much cheaper then the creation of a new process (via fork()). This would be well worth testing on some representative sample systems. Within the past year and a half at one of my gigs some coworkers did tests on various platforms (Irix, Solaris, a few variations of Linux and *BSDs) and concluded that in fact the threads implementations were often *slower* than using processes for moving and distributing the sorts of data that they were playing with. With copy-on-write and interprocess pipes that are roughly equivalent to memcpy() speeds it was determined for that application that the best way to split up tasks was fork()ing and dup(). As always, your mileage will vary, but the one thing that consistently amazes me on the Un*x like operating systems is that usually the programmatically simplest way to implement something has been optimized all to heck. A lesson that comes hard to those of us who grew up on MS systems. Dan
On Mon, Dec 04, 2000 at 03:17:00PM -0800, Adam Haberlach wrote: > Typically (on a well-written OS, at least), the spawning of a thread > is much cheaper then the creation of a new process (via fork()). Unless I'm mistaken, the back-end is only forked when starting a new connection, in which case the latency of doing the initial TCP tri-state and start-up queries is much larger than any process creation cost. On Linux 2.2.16 on a 500MHz PIII, I can do the fork/exit/wait sequence in about 164us. On the same server, I can make/break a PostgreSQL connection in about 19,000us (with 0% CPU idle, about 30% CPU system). Even if we can manage to get a thread for free, and assume that the fork from postmaster takes more than 164us, it won't make a big difference once the other latencies are worked out. > Also, since everything in a group of threads (I'll call 'em a team) Actually, you call them a process. That is the textbook definition. > shares the > same address space, there can be some memory overhead savings. Only slightly. All of the executable and libraries should already be shared, as will all non-modified data. If the data is modified by the threads, you'll need seperate copies for each thread anyways, so the net difference is small. I'm not denying there would be a difference. Compared to seperate processes, threads are more efficient. Doing a context switch between threads means there is no PTE invalidations, which makes them quicker than between processes. Creation would be a bit faster due to just linking in the VM to a new thread rather than marking it all as COW. The memory savings would come from reduced fragmentation of the modified data (if you have 1 byte modified on each of 100 pages, the thread would grow by a few K, compared to 400K for processes). I'm simply arguing that the differences don't appear to be significant compared to the other costs involved. -- Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/
*snip* > > > > Once all the questions regarding "why not" have been answered, it would > > be good to also ask "why use threads?" Do they simplify the code? Do > > they offer significant performance or efficiency gains? What do they > > give, other than being buzzword compliant? > The primary advantage that I see is that a single postgres process can benefit from multiple processors. I see little advantage to using thread for client connections.
On Mon, Dec 04, 2000 at 02:30:31PM -0800, Dan Lyke wrote:
> Adam Haberlach writes:
> > Typically (on a well-written OS, at least), the spawning of a thread
> > is much cheaper then the creation of a new process (via fork()).
> This would be well worth testing on some representative sample
> systems.
Using the following program for timing process creation and cleanup:
main() { int i; int pid; for (i=0; i<100000; ++i) { pid=fork(); if(pid==-1) exit(1); if(!pid) _exit(0);
waitpid(pid,0,0);} exit(0);
}
And using the following program for timing thread creation and cleanup:
#include <pthread.h>
threadfn() { pthread_exit(0); }
main() { int i; pthread_t thread; for (i=0; i<100000; ++i) { if (pthread_create(&thread, 0, threadfn, 0)) exit(1);
if(pthread_join(thread, 0)) exit(1); } exit(0);
}
On a relatively unloaded 500MHz PIII running Linux 2.2, the fork test
program took a minimum of 16.71 seconds to run (167us per
fork/exit/wait), and the thread test program took a minimum of 12.10
seconds to run (121us per pthread_create/exit/join). I use the minimums
because those would be the runs where the tasks were least interfered
with by other tasks. This amounts to a roughly 25% speed improvement
for threads over processes, for the null-process case.
If I add the following lines before the for loop: char* m; m=malloc(1024*1024); memset(m,0,1024,1024);
The cost for doing the fork balloons to 240us, whereas the cost for
doing the thread is constant. So, the cost of marking the pages as COW
is quite significant (using those numbers, 73us/MB).
So, forking a process with lots of data is expensive. However, most of
the PostgreSQL data is in a SysV IPC shared memory segment, which
shouldn't affect the fork numbers.
--
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/
Matthew wrote: > The primary advantage that I see is that a single postgres process > can benefit from multiple processors. I see little advantage to using thread > for client connections. Multiprocessors best benefit multiple backends. And the current forked model lends itself admirably to SMP. And I say that even after using a multithreaded webserver (AOLserver) for three and a half years. Of course, AOLserver also sanely uses the multi process PostgreSQL backends in a pooled fashion, but that's beside the point. -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
I would love to distribute this code to anybody who wants it. Any suggestions for a good place? However, calling the work a code redesign is a bit generous. This was more like a brute force hack. I just moved all the connection related global variables to a thread local "environment variable" and bypassed much of the postmaster code. I did this so I could port my app which was originally designed for Oracle OCI and Java. My app uses very few SQL statements but uses them over and over. I wanted true prepared statements linked to Java with JNI. I got both as well as batched transaction writes ( which was more relevant before WAL). In my situation, threads seemed much more flexible to implement, and I probably could not have done the port without it. Myron On Mon, 4 Dec 2000, Ross J. Reedstrom wrote: > Myron - > Putting aside the fork/threads discussion for a moment (the reasons, > both historical and other, such as inter-backend protection, are well > covered in the archives), the work you did sounds like an interesting > experiment in code redesign. Would you be willing to release the hacked > code somewhere for others to learn from? Hacking flex to generate > thread-safe code is of itself interesting, and the question about PG and > threads comes up so often, that an example of why it's not a simple task > would be useful. > > Ross >
Bruce Guenter <bruceg@em.ca> writes:
> [ some very interesting datapoints ]
>
> So, forking a process with lots of data is expensive. However, most of
> the PostgreSQL data is in a SysV IPC shared memory segment, which
> shouldn't affect the fork numbers.
I believe (but don't have numbers to prove it) that most of the present
backend startup time has *nothing* to do with thread vs process
overhead. Rather, the primary startup cost has to do with initializing
datastructures, particularly the system-catalog caches. A backend isn't
going to get much real work done until it's slurped in a useful amount
of catalog cache --- for example, until it's got the cache entries for
pg_class and the indexes thereon, it's not going to accomplish anything
at all.
Switching to a thread model wouldn't help this cost a bit, unless
we also switch to a shared cache model. That's not necessarily a win
when you consider the increased costs associated with cross-backend
or cross-thread synchronization needed to access or update the cache.
And if it *is* a win, we could get most of the same benefit in the
multiple-process model by keeping the cache in shared memory.
The reason that a new backend has to do all this setup work for itself,
rather than inheriting preloaded cache entries via fork/copy-on-write
from the postmaster, is that the postmaster isn't part of the ring of
processes that can access the database files directly. That was done
originally for robustness reasons: since the PM doesn't have to deal
with database access, cache invalidation messages, etc etc yadda yadda,
it is far simpler and less likely to crash than a real backend. If we
conclude that shared syscache is not a reasonable idea, it might be
interesting to look into making the PM into a full-fledged backend
that maintains a basic set of cache entries, so that these entries are
immediately available to new backends. But we'd have to take a real
hard look at the implications for system robustness/crash recovery.
In any case I think we're a long way away from the point where switching
to threads would make a big difference in connection startup time.
regards, tom lane
On Mon, 4 Dec 2000, Junfeng Zhang wrote: > All the major operating systems should have POSIX threads implemented. > Actually this can be configurable--multithreads or one thread. I don't understand this. The OS can be configured for one thread? How would that be any of use? > Thread-only server is unsafe, I agree. Maybe the following model can be a > little better. Several servers, each is multi-threaded. Every server can > support a maximum number of requests simultaneously. If anything bad > happends, it is limited to that server. There is no difference. If anything bad happens with the current multi-process server, all the postgres backends shutdown because the shared memory may be corrupted. > The cons side of processes model is not the startup time. It is about > kernel resource and context-switch cost. Processes consume much more > kernel resource than threads, and have a much higher cost for context > switch. The scalability of threads model is much better than that of > processes model. What kernel resources do a process use? There is some VM mapping overhead, a process table entry, and a file descriptor table. It is possible to support thousands of processes today. For instance, ftp.freesoftware.com supports up to 5000 FTP connections using a slightly modified ftpd (doesn't use inetd anymore). That means with 5000 users connected, that works out to 5000 processes active. Amazing but true. Some OSes (Linux is the main one) implement threads as pseudo processes. Linux threads are processes with a shared address space and file descriptor table. Context switch cost for threads can be lower if you are switching to a thread in the same process. That of course assumes that all context switches will occur within the same process, or the Linux everything-is-a-process model isn't used. > -Junfeng Tom
On Mon, Dec 04, 2000 at 08:43:24PM -0800, Tom Samplonius wrote: > Some OSes (Linux is the main one) implement threads as pseudo processes. > Linux threads are processes with a shared address space and file > descriptor table. > > Context switch cost for threads can be lower if you are switching to a > thread in the same process. That of course assumes that all context > switches will occur within the same process, or the Linux > everything-is-a-process model isn't used. Actually, context switch cost between threads is low on Linux as well, since the CPU's VM mappings don't get invalidated. This means that its page tables won't get reloaded, which is one of the large costs involved in context switches. Context switches between processes takes (with no significant VM) about 900 cycles (1.8us) on a 450MHz Celery. I would expect thread switch time to be slightly lower than that, and context switches between processes with large VMs would be much larger just due to the cost of reloading the page tables. -- Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/
> All the major operating systems should have POSIX threads implemented. > Actually this can be configurable--multithreads or one thread. > > Thread-only server is unsafe, I agree. Maybe the following model can be a > little better. Several servers, each is multi-threaded. Every server can > support a maximum number of requests simultaneously. If anything bad > happends, it is limited to that server. > > The cons side of processes model is not the startup time. It is about > kernel resource and context-switch cost. Processes consume much more > kernel resource than threads, and have a much higher cost for context > switch. The scalability of threads model is much better than that of > processes model. My question here is how much do we really context switch. We do quite a bit of work for each query, and I don't see us giving up the CPU very often, as would be the case for a GUI where each thread does a little work and goes to sleep. Also, as someone pointed out, the postmaster doesn't connect to the database, so there isn't much COW overhead. The big win is that all the text page are already in memory and shared by all backends. They can't modify those. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
-- Start of PGP signed section.
> On Mon, Nov 27, 2000 at 11:42:24PM -0600, Junfeng Zhang wrote:
> > I am new to postgreSQL. When I read the documents, I find out the Postmaster
> > daemon actual spawns a new backend server process to serve a new client
> > request. Why not use threads instead? Is that just for a historical reason,
> > or some performance/implementation concern?
>
> Once all the questions regarding "why not" have been answered, it would
> be good to also ask "why use threads?" Do they simplify the code? Do
> they offer significant performance or efficiency gains? What do they
> give, other than being buzzword compliant?
Good question. I have added this to the developers FAQ:
---------------------------------------------------------------------------
14) Why don't we use threads in the backend?
There are several reasons threads are not used:
Historically, threads were unsupported and buggy. An error in one backend can corrupt other backends. Speed
improvementsusing threads are small compared to the remaining backend startup time. The backend code would be
morecomplex.
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill,
Pennsylvania19026
> Adam Haberlach writes: > > Typically (on a well-written OS, at least), the spawning of a thread > > is much cheaper then the creation of a new process (via fork()). > > This would be well worth testing on some representative sample > systems. > > Within the past year and a half at one of my gigs some coworkers did > tests on various platforms (Irix, Solaris, a few variations of Linux > and *BSDs) and concluded that in fact the threads implementations were > often *slower* than using processes for moving and distributing the > sorts of data that they were playing with. > > With copy-on-write and interprocess pipes that are roughly equivalent > to memcpy() speeds it was determined for that application that the > best way to split up tasks was fork()ing and dup(). This brings up a good point. Threads are mostly useful when you have multiple processes that need to share lots of data, and the interprocess overhead is excessive. Because we already have that shared memory area, this benefit of threads doesn't buy us much. We sort of already have done the _shared_ part, and the addition of sharing our data pages is not much of a win. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> Bruce Guenter <bruceg@em.ca> writes: > > [ some very interesting datapoints ] > > > > So, forking a process with lots of data is expensive. However, most of > > the PostgreSQL data is in a SysV IPC shared memory segment, which > > shouldn't affect the fork numbers. > > I believe (but don't have numbers to prove it) that most of the present > backend startup time has *nothing* to do with thread vs process > overhead. Rather, the primary startup cost has to do with initializing > datastructures, particularly the system-catalog caches. A backend isn't > going to get much real work done until it's slurped in a useful amount > of catalog cache --- for example, until it's got the cache entries for > pg_class and the indexes thereon, it's not going to accomplish anything > at all. > > Switching to a thread model wouldn't help this cost a bit, unless > we also switch to a shared cache model. That's not necessarily a win > when you consider the increased costs associated with cross-backend > or cross-thread synchronization needed to access or update the cache. > And if it *is* a win, we could get most of the same benefit in the > multiple-process model by keeping the cache in shared memory. Of course, we would also have to know which database was being used next. Each database's system catalog can be different. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
> > On Mon, 4 Dec 2000, Junfeng Zhang wrote: > > > All the major operating systems should have POSIX threads implemented. > > Actually this can be configurable--multithreads or one thread. > > I don't understand this. The OS can be configured for one thread? How > would that be any of use? > > > Thread-only server is unsafe, I agree. Maybe the following model can be a > > little better. Several servers, each is multi-threaded. Every server can > > support a maximum number of requests simultaneously. If anything bad > > happends, it is limited to that server. > > There is no difference. If anything bad happens with the current > multi-process server, all the postgres backends shutdown because the > shared memory may be corrupted. Yes. Are we adding reliability with per-process backends. I think so because things are less likely to go haywire, and we are more likely to be able to clean things up in a failure. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <pgman@candle.pha.pa.us> writes:
>> There is no difference. If anything bad happens with the current
>> multi-process server, all the postgres backends shutdown because the
>> shared memory may be corrupted.
> Yes. Are we adding reliability with per-process backends.
Yes, we are: the postmaster forces a system-wide restart only if a
backend actually coredumps, or exits with elog(STOP). If a backend
curls up and dies with elog(FATAL), we assume it's a positive sign
that it was able to detect the error ;-), and keep plugging.
Now you might argue about whether any particular error case has been
misclassified, and I'm sure some are. But my point is that in a
multithread environment we couldn't risk allowing the other threads to
keep running after an elog(FATAL), either. Threads don't have *any*
protection against screwups in other threads.
A closely related point: how often do you hear of postmaster crashes?
They don't happen, as a rule. That's because the postmaster is (a)
simple and (b) decoupled from the backends. To convert backend process
launch into backend thread launch, the postmaster would have to live
in the same process space as the backends, and that means any backend
coredump would take down the postmaster too.
regards, tom lane
> Bruce Momjian <pgman@candle.pha.pa.us> writes: > >> There is no difference. If anything bad happens with the current > >> multi-process server, all the postgres backends shutdown because the > >> shared memory may be corrupted. > > > Yes. Are we adding reliability with per-process backends. > > Yes, we are: the postmaster forces a system-wide restart only if a > backend actually coredumps, or exits with elog(STOP). If a backend > curls up and dies with elog(FATAL), we assume it's a positive sign > that it was able to detect the error ;-), and keep plugging. It would be interesting to have one backend per database, and have all threads for that database running as threads. That would remove the startup problem, but each thread would have to have private copies of some system tuples it modifies. Another idea would be to have a file that holds the standard system tuples normally loaded. These files would sit in each database, and be read on startup. If one backend modifies one of these tuples, it can delete the file and have the next backend recreate it. We already have some of these for hard-wired tuples. This would improve startup time. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
For anyone interested, I have posted my multi-threaded version of PostgreSQL here. http://www.sacadia.com/mtpg.html It is based on 7.0.2 and the TAO CORBA ORB which is here. http://www.cs.wustl.edu/~schmidt/TAO.html Myron Scott mkscott@sacadia.com
On Mon, 1 Jan 2001, Myron Scott wrote: > For anyone interested, > > I have posted my multi-threaded version of PostgreSQL here. > > http://www.sacadia.com/mtpg.html How you solve locks? Via original IPC or you rewrite it to mutex (etc). Karel
spinlocks rewritten to mutex_ locktable uses sema_ some cond_ in bufmgr.c Myron Karel Zak wrote: > On Mon, 1 Jan 2001, Myron Scott wrote: > > >> For anyone interested, >> >> I have posted my multi-threaded version of PostgreSQL here. >> >> http://www.sacadia.com/mtpg.html > > > How you solve locks? Via original IPC or you rewrite it to mutex (etc). > > Karel
Myron Scott wrote: > > For anyone interested, > > I have posted my multi-threaded version of PostgreSQL here. > > http://www.sacadia.com/mtpg.html > > It is based on 7.0.2 and the TAO CORBA ORB which is here. > > http://www.cs.wustl.edu/~schmidt/TAO.html > > Myron Scott > mkscott@sacadia.com Sounds cool. Have you done any benchmarking? I would love to compare the threaded version against the process version and see if there is any difference. I suspect there will not be much, but I have been surprised more than once by unexpected behavior of IPC. I have not used Solaris threads, are they substantially different from pthreads? (Pthreads, on solaris, should be based on solaris threads, correct?) I would love to see the #ifdef __cplusplus put in all the postgres headers. -- http://www.mohawksoft.com
Karel Zak wrote: > On Tue, 2 Jan 2001, Myron Scott wrote:>>>> spinlocks rewritten to mutex_>> locktable uses sema_>> some cond_ in bufmgr.c>>>Interesting, have you some comperation between IPC PostgresSQl anf> your thread based PostgreSQL.>> Karel Yes, I did some comparisions but it is hard to make accurate evaluations on the data. I basically did 1000 inserts from 7.0.2 and the modified verison from 8 simultaneous clients. The original 7.0.2 was faster by an an order of magnitude. This needs to looked into more though. It was just a rough test b/c clients and server all running on the same machine (Ultra 10 w/512MB RAM). I don't really know what the impact of changing some of the locking mechanisms is. On the one hand, there is alot of overhead associated with using TAO ORB as the fe<->be protocol. The 7.0.2 fe<->be is pretty efficient, TAO with IIOP not as much so. At the same time, using prepared statements when doing the same insert with different variables over and over cuts out re-parsing and planning the statement on every execute. Lastly, I really didn't optimize my code at all. There are some places where GetEnv() in called over and over to get the thread local variable where it should only be called once in the method and reused. Speed wasn't the motivation, I just wanted to see if threads and PostgreSQL could be done. Myron
* Myron Scott <mscott@sacadia.com> [010102 07:45] wrote: > > Karel Zak wrote: > > > On Tue, 2 Jan 2001, Myron Scott wrote: > > > > > >> spinlocks rewritten to mutex_ > >> locktable uses sema_ > >> some cond_ in bufmgr.c > > > > > > Interesting, have you some comperation between IPC PostgresSQl anf > > your thread based PostgreSQL. > > > > Karel > > Yes, I did some comparisions but it is hard to > make accurate evaluations on the data. I basically > did 1000 inserts from 7.0.2 and the modified verison > from 8 simultaneous clients. The original 7.0.2 > was faster by an an order of magnitude. This needs to > looked into more though. It was just a rough test b/c clients > and server all running on the same machine (Ultra 10 > w/512MB RAM). It's possible what you're seeing is the entire process wait for a disk IO to complete. I'm wondering, how many lwps does your system use? Are all the threads bound to a single lwp or do you let the threads manager handle this all for you? > I don't really know what the impact of changing some of > the locking mechanisms is. heh. :) > On the one hand, there is alot of overhead associated > with using TAO ORB as the fe<->be protocol. The 7.0.2 > fe<->be is pretty efficient, TAO with IIOP not as much so. > At the same time, using prepared statements when doing > the same insert with different variables over and over > cuts out re-parsing and planning the statement on every > execute. > > Lastly, I really didn't optimize my code at all. There > are some places where GetEnv() in called over and over > to get the thread local variable where it should only > be called once in the method and reused. > > Speed wasn't the motivation, I just wanted to see if threads > and PostgreSQL could be done. Well it'd be better to see if threads+postgresql could be made faster than non-threaded postgresql. Do you see yourself with some extra free time in the upcommming weeks? :) -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk."
Alfred Perlstein wrote: > > It's possible what you're seeing is the entire process > wait for a disk IO to complete. > > I'm wondering, how many lwps does your system use? Are all > the threads bound to a single lwp or do you let the threads > manager handle this all for you? > Yeah, I looked at this. I have one thread per process that does all flushing of buffer pages at transaction commit. The client threads register buffer writes with this thread and wait for this thread to complete writes at transaction end. Unfortuately, selects also wait which really isn't nessessary. I hoped this would speed simultaneous connections. I created this as both a bound thread with its own lwp and a threads manager managed thread. I eventually settled on a threads manager managed thread, thinking that I wanted to set the priority of this thread low and commit as many transactions as possible simulateously. Maybe, I should rethink this. As for client threads, that is managed by TAO and I haven't really played with that configuration. Myron Scott mkscott@sacadia.com
* Myron Scott <mscott@sacadia.com> [010102 08:47] wrote: > > > Alfred Perlstein wrote: > > > > > > It's possible what you're seeing is the entire process > > wait for a disk IO to complete. > > > > I'm wondering, how many lwps does your system use? Are all > > the threads bound to a single lwp or do you let the threads > > manager handle this all for you? > > > > Yeah, I looked at this. I have one thread per process that > does all flushing of buffer pages at transaction commit. > The client threads register buffer writes with this thread and > wait for this thread to complete writes > at transaction end. Unfortuately, selects also wait which really > isn't nessessary. I hoped this would speed > simultaneous connections. I created this as both > a bound thread with its own lwp and a threads manager > managed thread. I eventually settled on a threads manager > managed thread, thinking that I wanted to set the priority > of this thread low and commit as many transactions as > possible simulateously. Maybe, I should rethink this. > > As for client threads, that is managed by TAO and I > haven't really played with that configuration. I think you're saying you have an lwp in what you used to be per-process context? Meaning if you have 4 connections, you should have 4 lwps doing disk IO? If that's true, most likely your problems may be elsewhere, but since you seem to be overcaching your writes you may be pessimizing the writes. I wouldn't try to buffer it all, instead I would hold off until a write is done that isn't contiguous to the last write position and then being a flush operation. -- -Alfred Perlstein - [bright@wintelcom.net|alfred@freebsd.org] "I have the heart of a child; I keep it in a jar on my desk."
I have put a new version of my multi-threaded postgresql experiment at http://www.sacadia.com/mtpg.html This one actually works. I have added a server based on omniORB, a CORBA 2.3 ORB from ATT. It is much smaller than TAO and uses the thread per connection model. I haven't added the java side of the JNI interface yet but the C++ side is there. It's still not stable but it is much better than the last. Myron Scott mkscott@sacadia.com
On Mon, 5 Feb 2001, Myron Scott wrote: > I have put a new version of my multi-threaded > postgresql experiment at > > http://www.sacadia.com/mtpg.html > > This one actually works. I have added a server > based on omniORB, a CORBA 2.3 ORB from ATT. It > is much smaller than TAO and uses the thread per > connection model. I haven't added the java side > of the JNI interface yet but the C++ side is there. > > It's still not stable but it is much better than > mkscott@sacadia.com Sorry I haven't time to see and test your experiment, but I have a question. How you solve memory management? The current mmgr is based on global variable CurrentMemoryContext that is very often changed and used.Use you for this locks? If yes it is probably problematic point for perfomance. Karel
> > Sorry I haven't time to see and test your experiment, > but I have a question. How you solve memory management? > The current mmgr is based on global variable > CurrentMemoryContext that is very often changed and used. > Use you for this locks? If yes it is probably problematic > point for perfomance. > > Karel > There are many many globals I had to work around including all the memory management stuff. I basically threw everything into and "environment" variable which I stored in a thread specific using thr_setspecific. Performance is acually very good for what I am doing. I was able to batch commit transactions which cuts down on fsync calls, use prepared statements from my client using CORBA, and the various locking calls for the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did some performance tests for inserts 20 clients, 900 inserts per client, 1 insert per transaction, 4 different tables. 7.0.2 About 10:52 average completion multi-threaded 2:42 average completion 7.1beta3 1:13 average completion If I increased the number of inserts per transaction, multi-threaded got closer to 7.1 for inserts. I haven't tested other other types of commands yet. Myron Scott mkscott@sacadia.com
On Tue, 6 Feb 2001, Myron Scott wrote: > There are many many globals I had to work around including all the memory > management stuff. I basically threw everything into and "environment" > variable which I stored in a thread specific using thr_setspecific. Yes, it's good. I working on multi-thread application server (http://mape.jcu.cz) and I use for this project some things from PG (like mmgr), I planning use same solution. > Performance is acually very good for what I am doing. I was able to batch > commit transactions which cuts down on fsync calls, use prepared > statements from my client using CORBA, and the various locking calls for > the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did > some performance tests for inserts > > 20 clients, 900 inserts per client, 1 insert per transaction, 4 different > tables. > > 7.0.2 About 10:52 average completion > multi-threaded 2:42 average completion > 7.1beta3 1:13 average completion It is very very good for time for 7.1, already look forward to 7.2! :-) BTW, I not sure if you anytime in future will see threads in official PostgreSQL and if you spending time on relevant things (IMHO). Karel
On Tue, 6 Feb 2001, Karel Zak wrote: > > On Tue, 6 Feb 2001, Myron Scott wrote: > > > There are many many globals I had to work around including all the memory > > management stuff. I basically threw everything into and "environment" > > variable which I stored in a thread specific using thr_setspecific. > > Yes, it's good. I working on multi-thread application server > (http://mape.jcu.cz) and I use for this project some things from PG (like > mmgr), I planning use same solution. > > > Performance is acually very good for what I am doing. I was able to batch > > commit transactions which cuts down on fsync calls, use prepared > > statements from my client using CORBA, and the various locking calls for > > the threads (cond_wait,mutex_lock, and sema_wait) seem pretty fast. I did > > some performance tests for inserts > > > > 20 clients, 900 inserts per client, 1 insert per transaction, 4 different > > tables. > > > > 7.0.2 About 10:52 average completion > > multi-threaded 2:42 average completion > > 7.1beta3 1:13 average completion > > It is very very good for time for 7.1, already look forward to 7.2! :-) > > BTW, I not sure if you anytime in future will see threads in > official PostgreSQL and if you spending time on relevant things (IMHO). There have been discussions about this, where we still do one process per client, but the backend of that process would use threads in order to improve performance on SMP boxes for that one client ...