Thread: Postgresql and multithreading
Is there any plans to make postgresql multithreading? Thanks in advance (and also for all who commented to my question regarding replication.) Anuradha NB: please don't open fire to declare war on whether multithreading is needed for PGSql or not. I am just expecting a black and white answer from the `authorities' ;) -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) If you can survive death, you can probably survive anything.
Anuradha Ratnaweera wrote: > > Is there any plans to make postgresql multithreading? > > Thanks in advance (and also for all who commented to my question > regarding replication.) > > Anuradha > > NB: please don't open fire to declare war on whether multithreading is > needed for PGSql or not. I am just expecting a black and white answer > from the `authorities' ;) We don't think it is needed, except perhaps for Win32 and Solaris, which have slow process creation times. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > > > Is there any plans to make postgresql multithreading? > > We don't think it is needed, except perhaps for Win32 and Solaris, which > have slow process creation times. Thanks, Bruce. But what I want to know is whether multithreading is likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? (as they did with Apache). Are there any plans to do so, or is postgres going to remain rather a multi-process application? Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) One nice thing about egotists: they don't talk about other people.
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > Is there any plans to make postgresql multithreading? > > Thanks in advance (and also for all who commented to my question > regarding replication.) > > Anuradha > > NB: please don't open fire to declare war on whether multithreading is > needed for PGSql or not. I am just expecting a black and white answer > from the `authorities' ;) This has been discussed at length in the past. The answer has always been no. Consult the archives for extensive heated discussion :-). Gavin
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 12:59:57AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > > Is there any plans to make postgresql multithreading? > > > > We don't think it is needed, except perhaps for Win32 and Solaris, which > > have slow process creation times. > > Thanks, Bruce. But what I want to know is whether multithreading is > likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? > (as they did with Apache). Are there any plans to do so, or is postgres > going to remain rather a multi-process application? It may be optional some day, most likely for Win32 at first, but we see little value to it on most other platforms; of course, we may be wrong. I am also not sure if it is a big win on Apache either; I think the jury is still out on that one, hence the slow adoption of 2.X, and we don't want to add threads and make a mess of the code or slow it down, which does often happen. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > > ... what I want to know is whether multithreading is likely to get > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. In that case, I wonder if it is worth folking a new project to add threading support to the backend? Of course, keeping in sync with the original would be lot of work. In that way, one should be able to test the hypothesis (whether threads improve things, or the other way round - if one likes it it that way :)) without messing around with stable postgres code, as they did and do with postgresql-R. And a minor question is wheter it is legal to keep the _changes_ in such a project GPL? > I am also not sure if it is a big win on Apache either; I think the > jury is still out on that one, hence the slow adoption of 2.X, As far as we are concened, it is the stability, rather than speed which still keeps us in 1.3. > and we don't want to add threads and make a mess of the code or slow > it down, which does often happen. Fully agreed. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) Equality is not when a female Einstein gets promoted to assistant professor; equality is when a female schlemiel moves ahead as fast as a male schlemiel. -- Ewald Nyquist
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. Probably not, but you can try. > In that way, one should be able to test the hypothesis (whether threads > improve things, or the other way round - if one likes it it that way :)) > without messing around with stable postgres code, as they did and do > with postgresql-R. I guess. > And a minor question is wheter it is legal to keep the _changes_ in such > a project GPL? We don't think we change the license, and we are happy with BSD. It certainly will never be merged in with a GPL, I can say that for sure. > > I am also not sure if it is a big win on Apache either; I think the > > jury is still out on that one, hence the slow adoption of 2.X, > > As far as we are concened, it is the stability, rather than speed which > still keeps us in 1.3. You could easily lose stability with threads -- don't think they are a free ride --- they aren't, and no, I don't feel like regurgitating what is already a 'thread' link on the TODO list. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. http://sourceforge.net/projects/mtpgsql > And a minor question is wheter it is legal to keep the _changes_ in such > a project GPL? Do you mean 'relicence the forked copy'? Gavin
On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > And a minor question is wheter it is legal to keep the _changes_ in such > > a project GPL? > > Do you mean 'relicence the forked copy'? Nope. To keep the `original' code licence as it is and to release the changes GPL? Is the question sane at first place? Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) You got to be very careful if you don't know where you're going, because you might not get there. -- Yogi Berra
Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > > > And a minor question is wheter it is legal to keep the _changes_ in such > > > a project GPL? > > > > Do you mean 'relicence the forked copy'? > > Nope. To keep the `original' code licence as it is and to release the > changes GPL? Is the question sane at first place? That would be a pretty big mess, I think. People would add your patch to our BSD code and it would be GPL. It could be done, of course. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Let me add one more thing on this "thread". This is one email in a long list of "Oh, gee, you aren't using that wizz-bang new sync/thread/aio/raid/raw feature" discussion where someone shows up and wants to know why. Does anyone know how to address these, efficiently? If we discuss it, it ends up causing a lot of effort on our part for the requestor to finally say, "Oh, gee, I didn't realize that." It isn't our job to explain that wizz-bang isn't always great. Maybe that should be our reply, "Wizz-bang isn't always great" and leave it at that. Of course, some will leave thinking we are just idiots, but then again, it takes one to know one. It is sort of like walking into a chess match and asking Bobby Fisher why he didn't move that pawn. :-) --------------------------------------------------------------------------- Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 03:40:47PM +1000, Gavin Sherry wrote: > > On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > > > > > And a minor question is wheter it is legal to keep the _changes_ in such > > > a project GPL? > > > > Do you mean 'relicence the forked copy'? > > Nope. To keep the `original' code licence as it is and to release the > changes GPL? Is the question sane at first place? > > Anuradha > > -- > > Debian GNU/Linux (kernel 2.4.18-xfs-1.1) > > You got to be very careful if you don't know where you're going, > because you might not get there. > -- Yogi Berra > > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote: > > Let me add one more thing on this "thread". This is one email in a > long list of "Oh, gee, you aren't using that wizz-bang new > sync/thread/aio/raid/raw feature" discussion where someone shows up > and wants to know why. Does anyone know how to address these, > efficiently? If somebody pops up asks such dumb questions without even looking at the FAQ, it is bad, if not idiotic, because it takes useful time away from the developers. But my question was not `why don't you implement this feature?`, but `do you have plans to implement this feature in the future?', and in the open source spirit of `if something is not there, go implement it yourself - without troubling developers' ;) Also, I have read the section 1.9 of the developers FAQ (Why don't we use threads in the backend?) long, long ago. > If we discuss it, it ends up causing a lot of effort on our part for > the requestor to finally say, "Oh, gee, I didn't realize that." Please don't. See the "NB" at end of my first mail of this thread. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) QOTD:"I'll listen to reason when it comes out on CD."
On 16 Oct 2002 at 1:25, Bruce Momjian wrote: > Anuradha Ratnaweera wrote: > > Thanks, Bruce. But what I want to know is whether multithreading is > > likely to get into in postgresql, say somewhere in 8.x, or even in 9.x? > > (as they did with Apache). Are there any plans to do so, or is postgres > > going to remain rather a multi-process application? > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. > I am also not sure if it is a big win on Apache either; I think the Well, I have done some stress testing on 1.3.26 and 2.0.39. Under same hardware and network setup and same test case, 1.3.26 maxed at 475-500 requests/sec and 2.0.39 gave flat 800 requests/sec. Yes, under light load, there is hardly any difference. But Apache2 series is definitely an improvement. > jury is still out on that one, hence the slow adoption of 2.X, and we > don't want to add threads and make a mess of the code or slow it down, > which does often happen. Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' syndrome, as far as I can see from news. Once linux distros start shipping with apache 2.x series *only*, the upgrade cycle will start rolling, I guess. ByeShridhar -- Programming Department: Mistakes made while you wait.
On 16 Oct 2002 at 15:40, Gavin Sherry wrote: > > In that case, I wonder if it is worth folking a new project to add > > threading support to the backend? Of course, keeping in sync with the > > original would be lot of work. > > http://sourceforge.net/projects/mtpgsql Last discussion that happened for multithreading was not to add threads per connection like mysql, but to split tasks between threads so that IO blocking and efficiently using SMP abilities would be taken care off IIRC.. One thread per connection isn't going to change much, at least for mainstream postgresql. For embedded, it might be a necessity.. ByeShridhar -- QOTD: "You're so dumb you don't even have wisdom teeth."
Bruce Momjian wrote: > Anuradha Ratnaweera wrote: <snip> > > Nope. To keep the `original' code licence as it is and to release the > > changes GPL? Is the question sane at first place? > > That would be a pretty big mess, I think. People would add your patch > to our BSD code and it would be GPL. It could be done, of course. Don't think so. The patches would be "derived code" that only exist because of the BSD licensed PostgreSQL base. Being "derived code" they'd have to be released as BSD and GPL wouldn't enter the picture, regardless if they're released separately as add-on patches or not. :-) Regards and best wishes, Justin Clift > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup. | Newtown Square, Pennsylvania 19073 -- "My grandfather once told me that there are two kinds of people: those who work and those who take the credit. He told me to try to be in the first group; there was less competition there." - Indira Gandhi
On Wed, 2002-10-16 at 01:27, Shridhar Daithankar wrote: > Well, slow adoption rate is attributed to 'apache 1.3.x is good enough for us' > syndrome, as far as I can see from news. Once linux distros start shipping with > apache 2.x series *only*, the upgrade cycle will start rolling, I guess. I think that's part of it. I think the other part is that by the time you're getting to huge r/s numbers, typical web site bandwidth is already used up. So, what's the point in adding more breathing room when you don't have the bandwidth to use it anyways. Greg
On Wed, 2002-10-16 at 04:34, Justin Clift wrote: > Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > <snip> > > > Nope. To keep the `original' code licence as it is and to release the > > > changes GPL? Is the question sane at first place? > > > > That would be a pretty big mess, I think. People would add your patch > > to our BSD code and it would be GPL. It could be done, of course. > > Don't think so. The patches would be "derived code" that only exist > because of the BSD licensed PostgreSQL base. > > Being "derived code" they'd have to be released as BSD and GPL wouldn't > enter the picture, regardless if they're released separately as add-on > patches or not. > I'm pretty sure BSD allows you to relicense derived code as you see fit. However, any derived project that was released GPL would have a hell of a time ever getting put back into the main source (short of relicensing). Robert Treat
On Wed, 2002-10-16 at 16:37, Robert Treat wrote: > I'm pretty sure BSD allows you to relicense derived code as you see fit. > However, any derived project that was released GPL would have a hell of > a time ever getting put back into the main source (short of > relicensing). Exactly. This is one of the reasons why BSD license is very much liked by proprietary software vendors (think MSFT), unlike the GPL which doesn't allow someone to change and redistribute their work with restrictive licenses. Cheers, Tycho (BTW: I'm not asking to change the license of Postgresql, I know the - dogmatic - answer to that one. So please don't misunderstand my mail) -- Tycho Fruru tycho@fruru.com "Prediction is extremely difficult. Especially about the future." - Niels Bohr
Bruce Momjian wrote: > Let me add one more thing on this "thread". This is one email in a long > list of "Oh, gee, you aren't using that wizz-bang new > sync/thread/aio/raid/raw feature" discussion where someone shows up and > wants to know why. Does anyone know how to address these, efficiently? You've brought up good points here. I'm sure that you consider me guilty of this with regard to my aio discussions. So I might offer a few specific suggestions. 1) Someone's taking the time to generate a summary of the current thinking with regard to a particular "whiz-bang" feature. - I can do this as a first pass for aio, if you think it's a good idea. 2) Including the pros and cons of the feature/implementation and how close the group is to deciding whether something would be worth doing. - I can also do this. 3) A set of criteria that would need to be met or demonstrated before a feature would be considered good enough for inclusion into the main code. If there was a separate section of the document "Developer's FAQ" that handled this, this would help. 4) A development/feature philosophy section. Maybe three or four paragraphs with one specific example: perhaps multi-threading. 5) I'd also suggest changing some specfics in the FAQ Section 1.2 where there is currently: * The usual process for source additions is: * * Review the TODO list. * Discuss hackers the desirability of the fix/feature. * How should it behave in complex circumstances? Add here that a check should be made to the new section in the FAQ described above. Also, section 1.1 has: * Discussion on the patch typically happens here. If the patch adds a * major feature, it would be a good idea to talk about it first on * the HACKERS list, in order to increase the chances of it being * accepted, as well as toavoid duplication of effort. We should perhaps add here a section describing the phenomenon you describe regarding "whiz-bang" features. I tried to prepare as best I could before bringing anything forward to HACKERS. In particular, I read the last two years of archives with anything to do with the WAL log and looked at the current code, read the TODOs, read a fair amount of discussions about aio. etc. So I was attempting to comply with my interpretation of the process. Yet, despite these efforts, you no doubt consider me guilty of creating unnecessary work, an outcome neither of us desired. I'm undeterred in my desire to come up with something meaningful and am working on some interesting tests. I do, however, now know that the level of scepticism and the standards of architectural purity are high. I consider this good all around. - Curtis
On Wed, 2002-10-16 at 23:08, Curtis Faith wrote: > Bruce Momjian wrote: > I tried to prepare as best I could before bringing anything forward to > HACKERS. In particular, I read the last two years of archives with anything > to do with the WAL log and looked at the current code, read the TODOs, read > a fair amount of discussions about aio. etc. So I was attempting to comply > with my interpretation of the process. Yet, despite these efforts, you no > doubt consider me guilty of creating unnecessary work, an outcome neither > of us desired. But that "unneccessary work" resulted in Tom finding and fixing an unintended behaviour (aka a performance bug) that prevented postgres from ever doing more than 1 commit per disk revolution on non-lieing SCSI disks ;) > I'm undeterred in my desire to come up with something meaningful and am > working on some interesting tests. I do, however, now know that the level > of scepticism and the standards of architectural purity are high. I > consider this good all around. I still have big expectations for use of aio, especially considering that at least for free OSes one is not forced to stop at the DB/OS boundary, but are free to go and improve the os side implementation as well if it is needed. But still some empirical tests are probably needed - if we can keep IO occupied for 99% in a meaningful way withou aio, then the time is probably better spent on something else ;) -------------------- Hannu
On Wed, Oct 16, 2002 at 02:08:21PM -0400, Curtis Faith wrote: > > 2) Including the pros and cons of the feature/implementation and how close > the group is to deciding whether something would be worth doing. - I can > also do this. The pros and cons of many such features have been discussed over the lists as well as on the FAQs. But the second matter, the group's likehood their implementation cannot always be deduced from those communications or from docs. Therefore suggested material into the FAQs are going to be extremely useful to like-to-be developers. They also would hopefully reduce unnecessary traffic on the list. Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) I have found little that is good about human beings. In my experience most of them are trash. -- Sigmund Freud
On Wed, 16 Oct 2002, Bruce Momjian wrote: > It may be optional some day, most likely for Win32 at first, but we see > little value to it on most other platforms; of course, we may be wrong. > I am also not sure if it is a big win on Apache either; I think the > jury is still out on that one, hence the slow adoption of 2.X, and we > don't want to add threads and make a mess of the code or slow it down, > which does often happen. Actually, Apache2 is both multi-process and multi-thread, even in the 'multi-thread' model ... I've played a bit with it, and it works at a sort of 'half way point' taking advantage of advantages of both ... in fact, I really wish someone would look at it seriously, since it mimics alot of what we do to start with ... old apache - one parent process (ie. postmaster) with child process (ie. postgres) actually handling the work new apache - one parent process (ie. postmaster) with child processes (ie. postgres) actually handling the work,with a twist .. each child process can handle x threaded processes so, in a heavily loaded web server, you might see 10 httpd processes running, each of which handling 15 threaded connections ... even getting away from multiple db connections per child process, I could see some other areas where multi-threading could be useful, assuming that my limited knowleddge of threading is remotely correct ... a big/cool one could be: distributed/clustered databases a database could be setup on multiple servers, where the tables are created as 'CREATE TABLE newtable ON SERVER serverb', so that when you connect to that table, the child process knows to auto-matically establish connections to the remote servers to pull data in this would also work for inter-database queries, that several ppl in the past have asked for
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Let me add one more thing on this "thread". This is one email in a long
> list of "Oh, gee, you aren't using that wizz-bang new
> sync/thread/aio/raid/raw feature" discussion where someone shows up and
> wants to know why. Does anyone know how to address these, efficiently?
Simple: respond to 'em all with a one-line answer: "convince us why we
should use it". The burden of proof always seems to fall on the wrong
end in these discussions.
regards, tom lane
On Wed, 16 Oct 2002, Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:25:23AM -0400, Bruce Momjian wrote: > > Anuradha Ratnaweera wrote: > > > > > ... what I want to know is whether multithreading is likely to get > > > into in postgresql, say somewhere in 8.x, or even in 9.x? > > > > It may be optional some day, most likely for Win32 at first, but we see > > little value to it on most other platforms; of course, we may be wrong. > > In that case, I wonder if it is worth folking a new project to add > threading support to the backend? Of course, keeping in sync with the > original would be lot of work. Actually, if you go through the archives, there has been talk about what would have to be done to the main source tree towards getting threading included ... as well as a lengthy discussion about the steps involved. the first and foremost issue that needs to be addressed is cleaning up the global variables that are used throughout ...
On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > Let me add one more thing on this "thread". This is one email in a long > > list of "Oh, gee, you aren't using that wizz-bang new > > sync/thread/aio/raid/raw feature" discussion where someone shows up and > > wants to know why. Does anyone know how to address these, efficiently? > > Simple: respond to 'em all with a one-line answer: "convince us why we > should use it". The burden of proof always seems to fall on the wrong > end in these discussions. > > regards, tom lane That may be easier said that done. If you don't know what the objections are, it's hard to argue your case. If you do know and understand the objections, chances are you already know the code very well and/or have the mailing lists for a very long time. This basically means, you don't want to hear from anyone unless they are "one" with the code. That seems and sounds very anti-open source. After it's all said and done, I think you guys are barking up the wrong tree. Open Source is all about sharing ideas. Many times I've seen ideas expressed here that were not exact hits yet help facilitate discussion, understanding on the topics in general and in some cases may even spur other ideas or associated code fixes/improvements. When I first started on this list, I was scolded rather harshly for not asking all of my questions on the list. Originally, I was told to ask reasonable questions so that everyone can learn. Now, it seems, that people don't want to answer questions at all as it's bothering the developers. Commonly asked items, such as threading, seems like they are being addressed rather well without core developer participation. Right now, I'm not seeing any down sides to what's currently in place. If the core developers still feel like they are spending more time then they like, then perhaps those that following the mailing list can step forward a little more to address general questions and defer when needed. The topic, such as threading, was previously addressed yet people still followed up on the topic. Perhaps those that don't want to be bothered should allow more time for others to address the topic and leave it alone once it has been addressed. That alone seems like it would be a huge time saver for the developers and a better use of resources. Greg
Greg Copeland <greg@copelandconsulting.net> writes:
> On Thu, 2002-10-17 at 22:20, Tom Lane wrote:
>> Simple: respond to 'em all with a one-line answer: "convince us why we
>> should use it". The burden of proof always seems to fall on the wrong
>> end in these discussions.
> ... Now, it seems, that
> people don't want to answer questions at all as it's bothering the
> developers.
Not at all. But rehashing issues that have been talked out repeatedly
is starting to bug some of us ;-). Perhaps the correct "standard
answer" is more like "this has been discussed before, please read the
list archives".
regards, tom lane
Tom Lane wrote: > Greg Copeland <greg@copelandconsulting.net> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". I need to add something to the developers FAQ on this. I will do it soon. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Fri, 2002-10-18 at 09:28, Tom Lane wrote: > Greg Copeland <greg@copelandconsulting.net> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". I agree. That sounds like a much more reasonable response. In fact, if you were to simply let the fledglings respond, it would completely take you guys out of the loop. Perhaps something like a Wiki or FAQ-O-Matic can be added whereby, the user base can help maintain it? That would seemingly help take some load off of Bruce too. Greg
On Fri, Oct 18, 2002 at 10:28:38AM -0400, Tom Lane wrote: > Greg Copeland <greg@copelandconsulting.net> writes: > > On Thu, 2002-10-17 at 22:20, Tom Lane wrote: > >> Simple: respond to 'em all with a one-line answer: "convince us why we > >> should use it". The burden of proof always seems to fall on the wrong > >> end in these discussions. > > > ... Now, it seems, that > > people don't want to answer questions at all as it's bothering the > > developers. > > Not at all. But rehashing issues that have been talked out repeatedly > is starting to bug some of us ;-). Perhaps the correct "standard > answer" is more like "this has been discussed before, please read the > list archives". Let me explain my posting which started this `thread': - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use threads (and many times it has been discussed on thelist). - The TODO list has an item `Experiment with multi-threaded backend' and points to a mailing list discussion about the implementationby Myron Scott. His final comment is that he didn't `gain much performance' and `ended up with some prettyunmanagable code'. He also says that he wouldn't `personally try this again ... but there probably was a better way'. - I was going through the TODO list, and was wondering if I should try on this. But before doing that, naturally, I wantedto figure out if any of the core developers themselves have any plans of doing it. Now, I am trying hard to figure out why this `are you going to do this? otherwise I can try it', type posting was not differentiated from numerous `why don't YOU implement this feature' type postings ;) Anuradha -- Debian GNU/Linux (kernel 2.4.18-xfs-1.1) "Life is too important to take seriously." -- Corky Siegel
Anuradha Ratnaweera <anuradha@lklug.pdn.ac.lk> writes: > Let me explain my posting which started this `thread': > - The developer's FAQ section 1.9 explains why PostgreSQL doesn't use > threads (and many times it has been discussed on the list). > - The TODO list has an item `Experiment with multi-threaded backend' and > points to a mailing list discussion about the implementation by Myron > Scott. His final comment is that he didn't `gain much performance' > and `ended up with some pretty unmanagable code'. He also says that > he wouldn't `personally try this again ... but there probably was a > better way'. > - I was going through the TODO list, and was wondering if I should try > on this. But before doing that, naturally, I wanted to figure out if > any of the core developers themselves have any plans of doing it. > Now, I am trying hard to figure out why this `are you going to do this? > otherwise I can try it', type posting was not differentiated from > numerous `why don't YOU implement this feature' type postings ;) Well, if you'd actually said the above, we'd probably have replied to the effect of "we still think it's an unpromising project, but try it if you like". By my reading, your earliest postings in this thread showed no sign of any familiarity at all with the history: http://archives.postgresql.org/pgsql-hackers/2002-10/msg00704.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00707.php http://archives.postgresql.org/pgsql-hackers/2002-10/msg00711.php and so you got the sort of response that's usually given to clueless newbies... regards, tom lane
On the recurring debate of threading vs. forking, I was giving it a fwe thoughts a few days ago, particularly with concern to Linux's memory model. On IA32 platforms with over 4 gigs of memory, any one process can only "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new process obviously would allow a person to utilize very copious quantities of memory, assuming that (a) they were dealing with concurrent PG sessions, and (b) PG had reason to use the memory. I'm not entirely clear on threading in Linux - would it provide the same benefits, or would it suddenly lock you into a 3-gig memory space? steve
On 18 Oct 2002 at 13:11, Steve Wolfe wrote: > > On the recurring debate of threading vs. forking, I was giving it a fwe > thoughts a few days ago, particularly with concern to Linux's memory model. > > On IA32 platforms with over 4 gigs of memory, any one process can only > "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new > process obviously would allow a person to utilize very copious quantities of > memory, assuming that (a) they were dealing with concurrent PG sessions, and > (b) PG had reason to use the memory. Well IIRC PG can not use more than 2Gigs of memory or 250K shared buffers (Unless you alter the buffer size itself). This does not become an issue in itself. > I'm not entirely clear on threading in Linux - would it provide the same > benefits, or would it suddenly lock you into a 3-gig memory space? Well, if you need to allocate 3Gig of memory to single process like postgresql, it's time to get a 64bit CPU. IIRC linux run on quite a few of them. HTH ByeShridhar -- QOTD: "Oh, no, no... I'm not beautiful. Just very, very pretty."
"Steve Wolfe" <nw@codon.com> writes: > On the recurring debate of threading vs. forking, I was giving it a fwe > thoughts a few days ago, particularly with concern to Linux's memory model. > > On IA32 platforms with over 4 gigs of memory, any one process can only > "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new > process obviously would allow a person to utilize very copious quantities of > memory, assuming that (a) they were dealing with concurrent PG sessions, and > (b) PG had reason to use the memory. > > I'm not entirely clear on threading in Linux - would it provide the same > benefits, or would it suddenly lock you into a 3-gig memory space? Linux threads are basically processes that share the same VM space, so you'd be limited to 3GB or whatever, since that's what a VM space can "see". -Doug
"D. Hageman" <dhageman@dracken.com> writes: > This in many ways is a bogus argument in that 1) postgresql runs on more > then just Linux and 2) amount of memmory that can be addressed by a > process is tunable up to the point that it reaches a hardware limitation. 1) The OP specifically asked about Linux threads. 2) True up to a point--Linux (and most other Unices) reserve some part of the VM address space for the kernel. On 64-bitthis is a non-issue, on 32-bit it's quite important now that you can put 4+GB in a machine. > It also should be noted that when a process reaches such a size that it > better have a good reason. Now let us do a gedanken experiment and say > you do have a good reason - fork a couple of these and your machine will > thrash like nothing else ... also that whole hardware limitation will come > into play more sooner then later ... True enough. The only real use I can see for gobs of memory on a 32-bit PAE machine with PG is to give each process its own big hunk of 'sortmem' for doing large sorts. If you have 64 GB in the machine setting 'sortmem' to 1GB or so starts to look reasonable... -Doug
This in many ways is a bogus argument in that 1) postgresql runs on more then just Linux and 2) amount of memmory that can be addressed by a process is tunable up to the point that it reaches a hardware limitation. It also should be noted that when a process reaches such a size that it better have a good reason. Now let us do a gedanken experiment and say you do have a good reason - fork a couple of these and your machine will thrash like nothing else ... also that whole hardware limitation will come into play more sooner then later ... On 21 Oct 2002, Doug McNaught wrote: > "Steve Wolfe" <nw@codon.com> writes: > > > On the recurring debate of threading vs. forking, I was giving it a fwe > > thoughts a few days ago, particularly with concern to Linux's memory model. > > > > On IA32 platforms with over 4 gigs of memory, any one process can only > > "see" up to 3 or 4 gigs of that. Having each postmaster fork off as a new > > process obviously would allow a person to utilize very copious quantities of > > memory, assuming that (a) they were dealing with concurrent PG sessions, and > > (b) PG had reason to use the memory. > > > > I'm not entirely clear on threading in Linux - would it provide the same > > benefits, or would it suddenly lock you into a 3-gig memory space? > > Linux threads are basically processes that share the same VM space, so > you'd be limited to 3GB or whatever, since that's what a VM space can > "see". > > -Doug > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > -- //========================================================\\ || D. Hageman <dhageman@dracken.com> || \\========================================================//
I have updated the developers FAQ item 1.9 to address this: http://developer.postgresql.org/readtext.php?src/FAQ/FAQ_DEV.html+Developers-FAQ --------------------------------------------------------------------------- Anuradha Ratnaweera wrote: > On Wed, Oct 16, 2002 at 01:51:28AM -0400, Bruce Momjian wrote: > > > > Let me add one more thing on this "thread". This is one email in a > > long list of "Oh, gee, you aren't using that wizz-bang new > > sync/thread/aio/raid/raw feature" discussion where someone shows up > > and wants to know why. Does anyone know how to address these, > > efficiently? > > If somebody pops up asks such dumb questions without even looking at the > FAQ, it is bad, if not idiotic, because it takes useful time away from > the developers. > > But my question was not `why don't you implement this feature?`, but `do > you have plans to implement this feature in the future?', and in the > open source spirit of `if something is not there, go implement it > yourself - without troubling developers' ;) > > Also, I have read the section 1.9 of the developers FAQ (Why don't we > use threads in the backend?) long, long ago. > > > If we discuss it, it ends up causing a lot of effort on our part for > > the requestor to finally say, "Oh, gee, I didn't realize that." > > Please don't. See the "NB" at end of my first mail of this thread. > > Anuradha > > -- > > Debian GNU/Linux (kernel 2.4.18-xfs-1.1) > > QOTD: > "I'll listen to reason when it comes out on CD." > > > ---------------------------(end of broadcast)--------------------------- > TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073