Thread: 8.0.X and the ARC patent
FYI, core has discussed the pending IBM ARC patent and the usage of those ideas in 8.0. Tom has found a 2Q cache algorithm that predates the ARC patent and is very similar to ARC. The major difference is that it doesn't auto-size the ARC sub-buffers. Core believes it is best to backpatch this 2Q algorithm into 8.0.X to avoid any possible patent problems if the patent is granted and enforced. We are testing the use of the 2Q code to see if it has any performance impact. The 8.0.X release that uses 2Q will have more extensive testing than a normal minor release. 8.1 will have a new cache algorithm that hopefully will remove the buffer contention problems experienced by SMP machines. For development, this means we will _not_ have a shortened, non-initdb 8.1 release but a regular release cycle with the typical big batch of features. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Might it be possible to contact IBM directly and ask if they will allow usage of the patent for PostgreSQL. They've let 500 patents for open source, maybe they'll give a write off for this as well. There is an advantage beyond just not having to re-write the code, but it would also be sort of an IBM blessing, great PR. I will be at "Linux World" and see if there is an IBM booth, maybe I can get some contact info. > FYI, core has discussed the pending IBM ARC patent and the usage of > those ideas in 8.0. > > Tom has found a 2Q cache algorithm that predates the ARC patent and is > very similar to ARC. The major difference is that it doesn't auto-size > the ARC sub-buffers. > > Core believes it is best to backpatch this 2Q algorithm into 8.0.X to > avoid any possible patent problems if the patent is granted and > enforced. > > We are testing the use of the 2Q code to see if it has any performance > impact. The 8.0.X release that uses 2Q will have more extensive testing > than a normal minor release. 8.1 will have a new cache algorithm that > hopefully will remove the buffer contention problems experienced by SMP > machines. > > For development, this means we will _not_ have a shortened, non-initdb > 8.1 release but a regular release cycle with the typical big batch of > features. > > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup. | Newtown Square, Pennsylvania > 19073 > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faq >
pgsql@mohawksoft.com wrote: > Might it be possible to contact IBM directly and ask if they will allow > usage of the patent for PostgreSQL. They've let 500 patents for open > source, maybe they'll give a write off for this as well. > > There is an advantage beyond just not having to re-write the code, but it > would also be sort of an IBM blessing, great PR. > > I will be at "Linux World" and see if there is an IBM booth, maybe I can > get some contact info. I doubt they will give us something that extends to companies that sell PostgreSQL so I don't see the point. --------------------------------------------------------------------------- > > > > FYI, core has discussed the pending IBM ARC patent and the usage of > > those ideas in 8.0. > > > > Tom has found a 2Q cache algorithm that predates the ARC patent and is > > very similar to ARC. The major difference is that it doesn't auto-size > > the ARC sub-buffers. > > > > Core believes it is best to backpatch this 2Q algorithm into 8.0.X to > > avoid any possible patent problems if the patent is granted and > > enforced. > > > > We are testing the use of the 2Q code to see if it has any performance > > impact. The 8.0.X release that uses 2Q will have more extensive testing > > than a normal minor release. 8.1 will have a new cache algorithm that > > hopefully will remove the buffer contention problems experienced by SMP > > machines. > > > > For development, this means we will _not_ have a shortened, non-initdb > > 8.1 release but a regular release cycle with the typical big batch of > > features. > > > > -- > > Bruce Momjian | http://candle.pha.pa.us > > pgman@candle.pha.pa.us | (610) 359-1001 > > + If your life is a hard drive, | 13 Roberts Road > > + Christ can be your backup. | Newtown Square, Pennsylvania > > 19073 > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 5: Have you checked our extensive FAQ? > > > > http://www.postgresql.org/docs/faq > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
> pgsql@mohawksoft.com wrote: >> Might it be possible to contact IBM directly and ask if they will allow >> usage of the patent for PostgreSQL. They've let 500 patents for open >> source, maybe they'll give a write off for this as well. >> >> There is an advantage beyond just not having to re-write the code, but >> it >> would also be sort of an IBM blessing, great PR. >> >> I will be at "Linux World" and see if there is an IBM booth, maybe I can >> get some contact info. > > I doubt they will give us something that extends to companies that sell > PostgreSQL so I don't see the point. Actually, I think that's wrong. IBM has been really gung-ho about Linux. Of course this is an obvious movement against Microsoft domination of the server market, but they have made some very strong open source statements and have release about 500 patents to open source projects. The current open source patents extend to companies that sell other products, why not PostgreSQL as well? There is a *LOT* of crap going on with patents, there are so many issues and motives that is hard to keep track of why who is doing what. My bet is that it is 50/50. It all depends if IBM wants to hurt Oracle more than it wants to defned DB2.
Bruce Momjian wrote: >pgsql@mohawksoft.com wrote: > > >>Might it be possible to contact IBM directly and ask if they will allow >>usage of the patent for PostgreSQL. They've let 500 patents for open >>source, maybe they'll give a write off for this as well. >> >>There is an advantage beyond just not having to re-write the code, but it >>would also be sort of an IBM blessing, great PR. >> >>I will be at "Linux World" and see if there is an IBM booth, maybe I can >>get some contact info. >> >> > >I doubt they will give us something that extends to companies that sell >PostgreSQL so I don't see the point. > > Also if I recall didn't Tom already have a patch ready to be tested for the q2 stuff? Sincerely, Joshua D. Drake >--------------------------------------------------------------------------- > > > > >> >> >>>FYI, core has discussed the pending IBM ARC patent and the usage of >>>those ideas in 8.0. >>> >>>Tom has found a 2Q cache algorithm that predates the ARC patent and is >>>very similar to ARC. The major difference is that it doesn't auto-size >>>the ARC sub-buffers. >>> >>>Core believes it is best to backpatch this 2Q algorithm into 8.0.X to >>>avoid any possible patent problems if the patent is granted and >>>enforced. >>> >>>We are testing the use of the 2Q code to see if it has any performance >>>impact. The 8.0.X release that uses 2Q will have more extensive testing >>>than a normal minor release. 8.1 will have a new cache algorithm that >>>hopefully will remove the buffer contention problems experienced by SMP >>>machines. >>> >>>For development, this means we will _not_ have a shortened, non-initdb >>>8.1 release but a regular release cycle with the typical big batch of >>>features. >>> >>>-- >>> Bruce Momjian | http://candle.pha.pa.us >>> pgman@candle.pha.pa.us | (610) 359-1001 >>> + If your life is a hard drive, | 13 Roberts Road >>> + Christ can be your backup. | Newtown Square, Pennsylvania >>>19073 >>> >>>---------------------------(end of broadcast)--------------------------- >>>TIP 5: Have you checked our extensive FAQ? >>> >>> http://www.postgresql.org/docs/faq >>> >>> >>> >>---------------------------(end of broadcast)--------------------------- >>TIP 7: don't forget to increase your free space map settings >> >> >> > > > -- Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC Postgresql support, programming shared hosting and dedicated hosting. +1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com PostgreSQL Replicator -- production quality replication for PostgreSQL
Attachment
Joshua D. Drake wrote: > Bruce Momjian wrote: > > >pgsql@mohawksoft.com wrote: > > > > > >>Might it be possible to contact IBM directly and ask if they will allow > >>usage of the patent for PostgreSQL. They've let 500 patents for open > >>source, maybe they'll give a write off for this as well. > >> > >>There is an advantage beyond just not having to re-write the code, but it > >>would also be sort of an IBM blessing, great PR. > >> > >>I will be at "Linux World" and see if there is an IBM booth, maybe I can > >>get some contact info. > >> > >> > > > >I doubt they will give us something that extends to companies that sell > >PostgreSQL so I don't see the point. > > > > > > Also if I recall didn't Tom already have a patch ready to be > tested for the q2 stuff? Yes, he does. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
pgsql@mohawksoft.com writes:
> Might it be possible to contact IBM directly and ask if they will allow
> usage of the patent for PostgreSQL. They've let 500 patents for open
> source, maybe they'll give a write off for this as well.
If there were hard evidence that the ARC algorithm was far better than
the alternatives, it might be worth going in that direction. But there
is no such evidence. Jan has retracted his original opinion that the
ARC code is a big improvement over what we had before, and I haven't
seen anyone else putting up benchmark numbers that say we need to keep
ARC.
regards, tom lane
Tom Lane wrote: > pgsql@mohawksoft.com writes: > > Might it be possible to contact IBM directly and ask if they will allow > > usage of the patent for PostgreSQL. They've let 500 patents for open > > source, maybe they'll give a write off for this as well. > > If there were hard evidence that the ARC algorithm was far better than > the alternatives, it might be worth going in that direction. But there > is no such evidence. Jan has retracted his original opinion that the > ARC code is a big improvement over what we had before, and I haven't > seen anyone else putting up benchmark numbers that say we need to keep > ARC. And ARC has locking requirements that will make it very hard to fix our SMP buffer management problems in 8.1. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> And ARC has locking requirements that will make it very hard to fix our
> SMP buffer management problems in 8.1.
I am working on a buffer manager rewrite using the BufMgrLock breakup
and "clock sweep" management algorithm we've been discussing. At the
moment it's passing the regression tests but I'm sure there's some bugs
left :-(. I just now tried it on the infamous context-swap-storm test
case using a 4-way machine at Red Hat. PG 8.0 shows 20K or more CS/sec
and under 30% CPU usage in this situation. The patch shows 99% CPU
utilization and about 200 CS/sec (which is about nil, because the
machine shows ~100 CS/sec with nothing running except vmstat).
Still to be determined: what we lose in extra I/O from the presumably
less efficient cache management; also what sort of slowdown occurs on
a single-CPU machine that isn't going to get any benefit from the
increased amount of lock management. But it looks promising.
regards, tom lane
pgsql@mohawksoft.com wrote: >>pgsql@mohawksoft.com wrote: >> >>>Might it be possible to contact IBM directly and ask if they will allow >>>usage of the patent for PostgreSQL. They've let 500 patents for open >>>source, maybe they'll give a write off for this as well. >>> >>>There is an advantage beyond just not having to re-write the code, but >>>it >>>would also be sort of an IBM blessing, great PR. >>> >>>I will be at "Linux World" and see if there is an IBM booth, maybe I can >>>get some contact info. >> >>I doubt they will give us something that extends to companies that sell >>PostgreSQL so I don't see the point. > > > Actually, I think that's wrong. IBM has been really gung-ho about Linux. > Of course this is an obvious movement against Microsoft domination of the > server market, but they have made some very strong open source statements > and have release about 500 patents to open source projects. > > The current open source patents extend to companies that sell other > products, why not PostgreSQL as well? > > There is a *LOT* of crap going on with patents, there are so many issues > and motives that is hard to keep track of why who is doing what. My bet is > that it is 50/50. It all depends if IBM wants to hurt Oracle more than it > wants to defned DB2. > Seeing up close what IBM is doing with Eclipse, I must concur with this. I would be surprised if they said no. IBM knows that making revenue from software licenses is becoming increasingly harder. My impression is that they want to become less dependent on that and instead become more services oriented. This patent was filed before they started to change direction big time. Personally, I don't think they would care about it at all now. So IMHO, ask IBM. It can't hurt. Regards, Thomas Hallgren
Tom Lane wrote: > Still to be determined: what we lose in extra I/O from the presumably > less efficient cache management; also what sort of slowdown occurs on > a single-CPU machine that isn't going to get any benefit from the > increased amount of lock management. But it looks promising. Yea, that was one of my questions --- the new buffer locking helps SMP, but how much does it hurt single-cpu machines? Do we need autodetection or a GUC to control SMP-beneficial locking? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> Still to be determined: what we lose in extra I/O from the presumably
>> less efficient cache management; also what sort of slowdown occurs on
>> a single-CPU machine that isn't going to get any benefit from the
>> increased amount of lock management. But it looks promising.
> Yea, that was one of my questions --- the new buffer locking helps SMP,
> but how much does it hurt single-cpu machines?
So far I've not been able to measure any consistent difference, but you
know how much I trust pgbench ;-). I hope that Mark Wong can give us
some results on the OSDL setup soon.
regards, tom lane
On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: > For development, this means we will _not_ have a shortened, non-initdb > 8.1 release but a regular release cycle with the typical big batch of > features. Might we set a rough date for Beta freeze for 8.1 then? September 30th 2005 ? I see only benefit from publishing a not-before date now. It's up to Core if it slips, but it'll really help with gaining funding if people can accurately determine whether or not features can be added for inclusion in the next release. There are lots of potential donors waiting, so lets give them some certainty about which release their payback will occur in.... Best Regards, Simon Riggs
Simon Riggs wrote: > On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: > > For development, this means we will _not_ have a shortened, non-initdb > > 8.1 release but a regular release cycle with the typical big batch of > > features. > > Might we set a rough date for Beta freeze for 8.1 then? > > September 30th 2005 ? > > I see only benefit from publishing a not-before date now. It's up to > Core if it slips, but it'll really help with gaining funding if people > can accurately determine whether or not features can be added for > inclusion in the next release. There are lots of potential donors > waiting, so lets give them some certainty about which release their > payback will occur in.... Yea, probably September, but you can't dump a huge feature on us in August either without having talked about it first, so knowing the date might not be that helpful. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Fri, 25 Feb 2005, Bruce Momjian wrote: > Simon Riggs wrote: >> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: >>> For development, this means we will _not_ have a shortened, non-initdb >>> 8.1 release but a regular release cycle with the typical big batch of >>> features. >> >> Might we set a rough date for Beta freeze for 8.1 then? >> >> September 30th 2005 ? >> >> I see only benefit from publishing a not-before date now. It's up to >> Core if it slips, but it'll really help with gaining funding if people >> can accurately determine whether or not features can be added for >> inclusion in the next release. There are lots of potential donors >> waiting, so lets give them some certainty about which release their >> payback will occur in.... > > Yea, probably September, but you can't dump a huge feature on us in > August either without having talked about it first, so knowing the date > might not be that helpful. I thought we were looking at a 12-18 month cycle for 8.1? Which would put beta around January '06, no? ---- Marc G. Fournier Hub.Org Networking Services (http://www.hub.org) Email: scrappy@hub.org Yahoo!: yscrappy ICQ: 7615664
Bruce Momjian wrote: >Simon Riggs wrote: > > >>On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: >> >> >>>For development, this means we will _not_ have a shortened, non-initdb >>>8.1 release but a regular release cycle with the typical big batch of >>>features. >>> >>> >>Might we set a rough date for Beta freeze for 8.1 then? >> >>September 30th 2005 ? >> >>I see only benefit from publishing a not-before date now. It's up to >>Core if it slips, but it'll really help with gaining funding if people >>can accurately determine whether or not features can be added for >>inclusion in the next release. There are lots of potential donors >>waiting, so lets give them some certainty about which release their >>payback will occur in.... >> >> > >Yea, probably September, but you can't dump a huge feature on us in >August either without having talked about it first, so knowing the date >might not be that helpful. > > Am I misunderstanding here? September 30th is 15 months after the feature freeze for 8.0. On that basis we might reasonably expect 8.1 to appear around April 2006. Is that what's intended? cheers andrew
On Fri, 2005-02-25 at 00:18 -0500, Bruce Momjian wrote: > Simon Riggs wrote: > > On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: > > > For development, this means we will _not_ have a shortened, non-initdb > > > 8.1 release but a regular release cycle with the typical big batch of > > > features. > > > > Might we set a rough date for Beta freeze for 8.1 then? > > > > September 30th 2005 ? > > > > I see only benefit from publishing a not-before date now. It's up to > > Core if it slips, but it'll really help with gaining funding if people > > can accurately determine whether or not features can be added for > > inclusion in the next release. There are lots of potential donors > > waiting, so lets give them some certainty about which release their > > payback will occur in.... > > Yea, probably September, but you can't dump a huge feature on us in > August either without having talked about it first, so knowing the date > might not be that helpful. > [Sorry, just brought this up as part of another more general thread.] That's fine. Just have two (rough dates): - Major Feature Design Freeze - June 2005 (or 2Q2005) Major features may require significant discussion, prototyping and performance testing before agreement to include. You are advised that major features presented for initial code review may not be accepted after this date. - Overall Feature Freeze - Sept 2005 (or 3Q2005) All code implementing new features, large or small, should be complete by this date. Features may be rejected if supporting tests and documentation are not provided. Best Regards, Simon Riggs
On Fri, 2005-02-25 at 02:47 -0400, Marc G. Fournier wrote: > On Fri, 25 Feb 2005, Bruce Momjian wrote: > > > Simon Riggs wrote: > >> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: > >>> For development, this means we will _not_ have a shortened, non-initdb > >>> 8.1 release but a regular release cycle with the typical big batch of > >>> features. > >> > >> Might we set a rough date for Beta freeze for 8.1 then? > >> > >> September 30th 2005 ? > >> > >> I see only benefit from publishing a not-before date now. It's up to > >> Core if it slips, but it'll really help with gaining funding if people > >> can accurately determine whether or not features can be added for > >> inclusion in the next release. There are lots of potential donors > >> waiting, so lets give them some certainty about which release their > >> payback will occur in.... > > > > Yea, probably September, but you can't dump a huge feature on us in > > August either without having talked about it first, so knowing the date > > might not be that helpful. > > I thought we were looking at a 12-18 month cycle for 8.1? Which would put > beta around January '06, no? I'm happy with Core taking this decision, I'd just like to know +/- 1 month when that date is, i.e. which quarter of which year. Best Regards, Simon Riggs
Marc G. Fournier wrote: > On Fri, 25 Feb 2005, Bruce Momjian wrote: > > > Simon Riggs wrote: > >> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote: > >>> For development, this means we will _not_ have a shortened, non-initdb > >>> 8.1 release but a regular release cycle with the typical big batch of > >>> features. > >> > >> Might we set a rough date for Beta freeze for 8.1 then? > >> > >> September 30th 2005 ? > >> > >> I see only benefit from publishing a not-before date now. It's up to > >> Core if it slips, but it'll really help with gaining funding if people > >> can accurately determine whether or not features can be added for > >> inclusion in the next release. There are lots of potential donors > >> waiting, so lets give them some certainty about which release their > >> payback will occur in.... > > > > Yea, probably September, but you can't dump a huge feature on us in > > August either without having talked about it first, so knowing the date > > might not be that helpful. > > I thought we were looking at a 12-18 month cycle for 8.1? Which would put > beta around January '06, no? Oh, yea. I didn't realize people agreed with that, but I certainly agree. So that would put us at January. Simon, I think the major issue is how much does it help us to fix a date now vs. what is the benefit to allowing us to state it later. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
"Marc G. Fournier" <scrappy@postgresql.org> writes:
>> Yea, probably September, but you can't dump a huge feature on us in
>> August either without having talked about it first, so knowing the date
>> might not be that helpful.
> I thought we were looking at a 12-18 month cycle for 8.1? Which would put
> beta around January '06, no?
Although we've dropped the idea of letting the ARC problem drive a very
short 8.1 cycle, I would still like to see us shoot for a relatively
short 8.1 cycle --- less than a year for sure. The main reason is that
I think we'll be flushing out performance and feature issues in the
Windows port that we cannot reasonably back-patch into 8.0.*. PITR also.
In general it seems to me that 8.1 will need to have a consolidation and
fill-in-the-blanks flavor after what we did for 8.0, and that will be
helped by a shorter devel cycle.
As a proposal: feature freeze maybe early summer (June or July), beta
maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with
a good tailwind).
regards, tom lane
Tom Lane wrote: >Although we've dropped the idea of letting the ARC problem drive a very >short 8.1 cycle, I would still like to see us shoot for a relatively >short 8.1 cycle --- less than a year for sure. The main reason is that >I think we'll be flushing out performance and feature issues in the >Windows port that we cannot reasonably back-patch into 8.0.*. PITR also. >In general it seems to me that 8.1 will need to have a consolidation and >fill-in-the-blanks flavor after what we did for 8.0, and that will be >helped by a shorter devel cycle. > >As a proposal: feature freeze maybe early summer (June or July), beta >maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with >a good tailwind). > That sounds good. I would think that lots of users probably won't use the Windows port in production until 8.1 (performance reasons, paranoia etc...) I would hate to make put such a long delay in their adoption plans. One thing to consider while discussing the length of the cycle is what features are people planning on putting in? The 8.0 cycle had to be long due to the many huge improvements. I'm not aware of any 8.1 plans that are that ambitious, so why plan a long cycle when there are no features requiring it? Am I missing something? Matthew
Tom Lane wrote: > "Marc G. Fournier" <scrappy@postgresql.org> writes: > >> Yea, probably September, but you can't dump a huge feature on us in > >> August either without having talked about it first, so knowing the date > >> might not be that helpful. > > > I thought we were looking at a 12-18 month cycle for 8.1? Which would put > > beta around January '06, no? > > Although we've dropped the idea of letting the ARC problem drive a very > short 8.1 cycle, I would still like to see us shoot for a relatively > short 8.1 cycle --- less than a year for sure. The main reason is that > I think we'll be flushing out performance and feature issues in the > Windows port that we cannot reasonably back-patch into 8.0.*. PITR also. > In general it seems to me that 8.1 will need to have a consolidation and > fill-in-the-blanks flavor after what we did for 8.0, and that will be > helped by a shorter devel cycle. > > As a proposal: feature freeze maybe early summer (June or July), beta > maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with > a good tailwind). That is fine with me too. Let's see how mush 8.0 fixing we need in 8.1 and that will help determine the cutoff date, as will completed features that we want to get into a public release. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
"Matthew T. O'Connor" <matthew@zeut.net> writes:
> One thing to consider while discussing the length of the cycle is what
> features are people planning on putting in? The 8.0 cycle had to be
> long due to the many huge improvements. I'm not aware of any 8.1 plans
> that are that ambitious, so why plan a long cycle when there are no
> features requiring it? Am I missing something?
The subtext here is that people are trying to decide what they intend to
shoot for in this cycle, and so they are asking Core what the schedule
target is. You really misunderstand the dynamics. 8.0 didn't start out
to be what it ended up being; there was no master plan, and never has
been. The most organization we've ever had is for Core to set a feature
freeze target date well in advance.
regards, tom lane
On Sat, Feb 19, 2005 at 12:57:52AM -0500, Tom Lane wrote: > So far I've not been able to measure any consistent difference, but you > know how much I trust pgbench ;-). I hope that Mark Wong can give us > some results on the OSDL setup soon. Sorry for the delay, broken laptop, vacation, etc. Here's a baseline against 8.0.1: http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/ throughput 3639.97 Against CVS from 20050217 and the 2Q patch: http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/ throughput3875.84 About a 6% increase, but if you look at the performance over time, it's degrading steadily. The latter throughput peaks near 5000. Mark
Mark Wong <markw@osdl.org> writes:
> About a 6% increase, but if you look at the performance over time,
> it's degrading steadily. The latter throughput peaks near 5000.
Curious. The immediate question is "does it ever flatten out, and
if so at what TPM rate compared to 8.0.1?" Could you run the same
test for a longer duration?
Also, I suppose the large spike at 30 minutes is from a checkpoint?
regards, tom lane
On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: > Mark Wong <markw@osdl.org> writes: > > About a 6% increase, but if you look at the performance over time, > > it's degrading steadily. The latter throughput peaks near 5000. > > Curious. The immediate question is "does it ever flatten out, and > if so at what TPM rate compared to 8.0.1?" Could you run the same > test for a longer duration? The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q patch? I can run a longer duration and see how it looks. > Also, I suppose the large spike at 30 minutes is from a checkpoint? Yes, checkpoint_timeout is set to 1800 seconds. Mark
Mark Wong <markw@osdl.org> writes:
> On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
>> Curious. The immediate question is "does it ever flatten out, and
>> if so at what TPM rate compared to 8.0.1?" Could you run the same
>> test for a longer duration?
> The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
> patch? I can run a longer duration and see how it looks.
My point was that unpatched 8.0.1 seems to have a pretty level TPM
rate. If the patched version levels out at something not far below
that, I'll be satisfied. If it continues to degrade then I won't be
satisfied ... but the test stops short of telling what will happen.
If you could run it for 2 hours then we'd probably know enough.
regards, tom lane
On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote: > Mark Wong <markw@osdl.org> writes: > > On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: > >> Curious. The immediate question is "does it ever flatten out, and > >> if so at what TPM rate compared to 8.0.1?" Could you run the same > >> test for a longer duration? > > > The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q > > patch? I can run a longer duration and see how it looks. > > My point was that unpatched 8.0.1 seems to have a pretty level TPM > rate. If the patched version levels out at something not far below > that, I'll be satisfied. If it continues to degrade then I won't be > satisfied ... but the test stops short of telling what will happen. > If you could run it for 2 hours then we'd probably know enough. Ah, ok. I've reapplied the 2Q patch to CVS from 20050301:http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ I ran it for 3 hours, just in case, and the charts suggest it flattens out after 2 hours. Mark
I was just looking at the config parameters, and you have the shared buffers set to 60k, and the effective cache set to 1k ???? Dave Mark Wong wrote: >On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote: > > >>Mark Wong <markw@osdl.org> writes: >> >> >>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: >>> >>> >>>>Curious. The immediate question is "does it ever flatten out, and >>>>if so at what TPM rate compared to 8.0.1?" Could you run the same >>>>test for a longer duration? >>>> >>>> >>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q >>>patch? I can run a longer duration and see how it looks. >>> >>> >>My point was that unpatched 8.0.1 seems to have a pretty level TPM >>rate. If the patched version levels out at something not far below >>that, I'll be satisfied. If it continues to degrade then I won't be >>satisfied ... but the test stops short of telling what will happen. >>If you could run it for 2 hours then we'd probably know enough. >> >> > >Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > >I ran it for 3 hours, just in case, and the charts suggest it flattens >out after 2 hours. > >Mark > >---------------------------(end of broadcast)--------------------------- >TIP 7: don't forget to increase your free space map settings > > > > -- Dave Cramer http://www.postgresintl.com 519 939 0336 ICQ#14675561
Yes, those parameters are based on a series of test results here:http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html Run 264 provided the best results, so I'm trying to continue with the database parameters used there. Mark On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote: > I was just looking at the config parameters, and you have the shared > buffers set to 60k, and the effective cache set to 1k ???? > > Dave > > Mark Wong wrote: > > >On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote: > > > > > >>Mark Wong <markw@osdl.org> writes: > >> > >> > >>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: > >>> > >>> > >>>>Curious. The immediate question is "does it ever flatten out, and > >>>>if so at what TPM rate compared to 8.0.1?" Could you run the same > >>>>test for a longer duration? > >>>> > >>>> > >>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q > >>>patch? I can run a longer duration and see how it looks. > >>> > >>> > >>My point was that unpatched 8.0.1 seems to have a pretty level TPM > >>rate. If the patched version levels out at something not far below > >>that, I'll be satisfied. If it continues to degrade then I won't be > >>satisfied ... but the test stops short of telling what will happen. > >>If you could run it for 2 hours then we'd probably know enough. > >> > >> > > > >Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > > > >I ran it for 3 hours, just in case, and the charts suggest it flattens > >out after 2 hours. > > > >Mark
OK. I doubt that it impacts the results of the particular test, but it is non-intuitive (in my mind at least) Did you change anything else between 263 and 264? From the table it appears that you are changing vm parameters as well as database configuration parameters between runs ? Dave Mark Wong wrote: >Yes, those parameters are based on a series of test results here: > http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html > >Run 264 provided the best results, so I'm trying to continue with the >database parameters used there. > >Mark > >On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote: > > >>I was just looking at the config parameters, and you have the shared >>buffers set to 60k, and the effective cache set to 1k ???? >> >>Dave >> >>Mark Wong wrote: >> >> >> >>>On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote: >>> >>> >>> >>> >>>>Mark Wong <markw@osdl.org> writes: >>>> >>>> >>>> >>>> >>>>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: >>>>> >>>>> >>>>> >>>>> >>>>>>Curious. The immediate question is "does it ever flatten out, and >>>>>>if so at what TPM rate compared to 8.0.1?" Could you run the same >>>>>>test for a longer duration? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q >>>>>patch? I can run a longer duration and see how it looks. >>>>> >>>>> >>>>> >>>>> >>>>My point was that unpatched 8.0.1 seems to have a pretty level TPM >>>>rate. If the patched version levels out at something not far below >>>>that, I'll be satisfied. If it continues to degrade then I won't be >>>>satisfied ... but the test stops short of telling what will happen. >>>>If you could run it for 2 hours then we'd probably know enough. >>>> >>>> >>>> >>>> >>>Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: >>> http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ >>> >>>I ran it for 3 hours, just in case, and the charts suggest it flattens >>>out after 2 hours. >>> >>>Mark >>> >>> > > > > -- Dave Cramer http://www.postgresintl.com 519 939 0336 ICQ#14675561
On Wed, Mar 02, 2005 at 07:21:41AM -0800, Mark Wong wrote: > Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > > I ran it for 3 hours, just in case, and the charts suggest it flattens > out after 2 hours. Looking at the "Response Time Charts" 8.0.1/ARC http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html 20050301 with 2Q patch http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html It seems like the average response time has gone down, but the worse case ceiling has raised about 35%. -Mike
Mark Wong <markw@osdl.org> writes: > Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > I ran it for 3 hours, just in case, and the charts suggest it flattens > out after 2 hours. Thanks. This seems odd though, since it appears to level out at something above 4K TPM. Your previous run http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/ shows it dropping to 3500 or so. What changed? regards, tom lane
On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote: > Mark Wong <markw@osdl.org> writes: > > Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > > > I ran it for 3 hours, just in case, and the charts suggest it flattens > > out after 2 hours. > > Thanks. This seems odd though, since it appears to level out at > something above 4K TPM. Your previous run > http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/ > shows it dropping to 3500 or so. What changed? Other than pulling from CVS at a different time, it should all be the same parameters, etc. Mark
Between those two runs, the vm parameters are the same. Mark On Wed, Mar 02, 2005 at 11:04:21AM -0500, Dave Cramer wrote: > OK. I doubt that it impacts the results of the particular test, but it > is non-intuitive (in my mind at least) > Did you change anything else between 263 and 264? From the table it > appears that you are changing vm parameters > as well as database configuration parameters between runs ? > > Dave > > Mark Wong wrote: > > >Yes, those parameters are based on a series of test results here: > > http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html > > > >Run 264 provided the best results, so I'm trying to continue with the > >database parameters used there. > > > >Mark > > > >On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote: > > > > > >>I was just looking at the config parameters, and you have the shared > >>buffers set to 60k, and the effective cache set to 1k ???? > >> > >>Dave > >> > >>Mark Wong wrote: > >> > >> > >> > >>>On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote: > >>> > >>> > >>> > >>> > >>>>Mark Wong <markw@osdl.org> writes: > >>>> > >>>> > >>>> > >>>> > >>>>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>Curious. The immediate question is "does it ever flatten out, and > >>>>>>if so at what TPM rate compared to 8.0.1?" Could you run the same > >>>>>>test for a longer duration? > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q > >>>>>patch? I can run a longer duration and see how it looks. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>My point was that unpatched 8.0.1 seems to have a pretty level TPM > >>>>rate. If the patched version levels out at something not far below > >>>>that, I'll be satisfied. If it continues to degrade then I won't be > >>>>satisfied ... but the test stops short of telling what will happen. > >>>>If you could run it for 2 hours then we'd probably know enough. > >>>> > >>>> > >>>> > >>>> > >>>Ah, ok. I've reapplied the 2Q patch to CVS from 20050301: > >>> http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/ > >>> > >>>I ran it for 3 hours, just in case, and the charts suggest it flattens > >>>out after 2 hours. > >>> > >>>Mark > >>> > >>> > > > > > > > > > > -- > Dave Cramer > http://www.postgresintl.com > 519 939 0336 > ICQ#14675561 -- Mark Wong - - markw@osdl.org Open Source Development Lab Inc - A non-profit corporation 12725 SW Millikan Way - Suite 400 - Beaverton, OR 97005 (503) 626-2455 (office) (503) 626-2436 (fax) http://developer.osdl.org/markw/
Mark Wong <markw@osdl.org> writes: > On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote: >> Thanks. This seems odd though, since it appears to level out at >> something above 4K TPM. Your previous run >> http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/ >> shows it dropping to 3500 or so. What changed? > Other than pulling from CVS at a different time, it should all be > the same parameters, etc. Hmph. The truth is probably somewhere in between these two curves. But in any case, I think we can make the conclusion we wanted to make: 2Q isn't seriously worse than ARC. Since this is a dead line of development anyway in view of the early results with the clock sweep algorithm, I don't think there's any need to spend more time measuring the differences carefully. I'll go ahead and apply the 2Q patch to the 8.0 branch, unless there are objections? regards, tom lane
Michael Adler <adler@pobox.com> writes: > Looking at the "Response Time Charts" > 8.0.1/ARC > http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html > 20050301 with 2Q patch > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html > It seems like the average response time has gone down, but the worse > case ceiling has raised about 35%. The worst cases are associated with checkpoints. I'm not sure why a checkpoint would have a greater effect on the 2Q system than an ARC system --- checkpoint doesn't request any new buffers so you'd think it'd be independent. Maybe this says that the bgwriter is less effective with 2Q, so that there are more dirty buffers remaining to be written at the checkpoint? But why? regards, tom lane
Tom Lane wrote: > Mark Wong <markw@osdl.org> writes: > > On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote: > >> Thanks. This seems odd though, since it appears to level out at > >> something above 4K TPM. Your previous run > >> http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/ > >> shows it dropping to 3500 or so. What changed? > > > Other than pulling from CVS at a different time, it should all be > > the same parameters, etc. > > Hmph. The truth is probably somewhere in between these two curves. > But in any case, I think we can make the conclusion we wanted to > make: 2Q isn't seriously worse than ARC. Since this is a dead line > of development anyway in view of the early results with the clock > sweep algorithm, I don't think there's any need to spend more time > measuring the differences carefully. He reported a huge benefit in current CVS, like 30% --- was that because of the clock algorithm? > I'll go ahead and apply the 2Q patch to the 8.0 branch, unless there > are objections? Good. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Tom, I should be able to run more OLTP benchmarks, and a DSS benchmark, within the next week. Please wait until I complete those before considering an 8.0.2 release with the new code. The machine and setup Mark is testing on is kind of end-of-the-curve; I have some more middle-of-the-road machines to test. I also want to run one 6-hour or 12-hour test to see if the flattening out is actually flat. -- Josh Berkus Aglio Database Solutions San Francisco
Dave Cramer <pg@fastcrypt.com> writes: > I was just looking at the config parameters, and you have the shared buffers > set to 60k, and the effective cache set to 1k ???? I was actually going to suggest that the performance degradation might be because of an excessively high shared_buffers setting. That was before I saw this comment. The only reason I could imagine the performance degradation would be because more and more CPU time is being spent traversing the 2Q LRU buffer lists. I would try it with a shared buffer setting of 10k to see if it levels out sooner at a higher TPM. I would also suggest setting checkpoint_timeout to something more realistic. All your 60m tests that show a single checkpoint in the middle are being deceptive since half the data in the test hasn't even been checkpointed. You should have enough checkpoints in your test that they're represented in the results realistically. If you want 60m to be a reasonably representative sample then I would suggest a checkpoint_timeout of 300-600 (ie, checkpoints every 5-10m) so you get 10-20 checkpoints in the result. And so that a maximum of 5-10% of the data isn't being checkpointed in the test. That would also make those huge performance dropouts a little less dramatic. And it might give us a chance to see how effective the bgwriter is at smoothing them out. Personally, as a user, I think it's more important to look at the maximum transaction latency than the average throughput. -- greg
Josh Berkus <josh@agliodbs.com> writes:
> I should be able to run more OLTP benchmarks, and a DSS benchmark, within the
> next week. Please wait until I complete those before considering an 8.0.2
> release with the new code.
Sure, there's no hurry to push out 8.0.2 (and we need to have some beta
testing done on it anyway). I have committed the proposed patch into
the REL8_0_STABLE branch, so you can just pull the branch tip from CVS
to do testing.
regards, tom lane
On Wed, 2005-03-02 at 20:55 -0500, Tom Lane wrote: > Michael Adler <adler@pobox.com> writes: > > Looking at the "Response Time Charts" > > > 8.0.1/ARC > > http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html > > > 20050301 with 2Q patch > > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html > > > It seems like the average response time has gone down, but the worse > > case ceiling has raised about 35%. > > The worst cases are associated with checkpoints. I'm not sure why a > checkpoint would have a greater effect on the 2Q system than an ARC > system --- checkpoint doesn't request any new buffers so you'd think > it'd be independent. Maybe this says that the bgwriter is less > effective with 2Q, so that there are more dirty buffers remaining to > be written at the checkpoint? But why? The pattern seems familiar. Reduced average response time increases total throughput, which on this test means we have more dirty buffers to write at checkpoint time. I would not neccessarily suspect 2Q over ARC, at least initially. The pattern of behaviour is similar across ARC, 2Q and Clock, though the checkpoint points differ in intensity. The latter makes me suspect BufMgrLock contention or similar. There is a two-level effect at Checkpoint time...first we have the write from PostgreSQL buffers to OS cache, then we have the write from OS cache to disk by the pdflush daemon. At this point, I'm not certain whether the delay is caused by the checkpointing or the pdflush daemons. Mark and I had discussed some investigations around that. This behaviour is new in the 2.6 kernel, so it is possible there is an unpleasant interaction there, though I do not wish to cast random blame. Checkpoint doesn't request new buffers, but it does require the BufMgrLock in order to write all of the dirty buffers. It could be that the I/Os map direct to OS cache, so that the tight loop to write out dirty buffers causes such an extreme backlog for the BufMgrLock that it takes more than a minute to clear and return to normal contention. It could be that at checkpoint time, the number of writes exceeds the dirty_ratio and the kernel forces the checkpoint process to bypass the cache and pdflush daemons altogether, and performing the I/O itself. Single-threaded, this would display the scalability profile we see. Some kernel level questions in there... There is no documented event-state model for LWlock acquisition, so it might be possible that there is a complex bottleneck in amongst them. Amdahl's Law tells me that looking at the checkpoints is the next best action for tuning, since they add considerably to the average response time. Looking at the oprofile for the run as a whole is missing out the delayed transaction behaviour that occurs during checkpoints. I would like to try and catch an oprofile of the system while performing a checkpoint, as a way to give us some clues. Perhaps that could be achieved by forcing a manual checkpoint as superuser, and making that interaction cause a switch to a new oprofile output file. Best Regards, Simon Riggs
Simon Riggs <simon@2ndquadrant.com> writes:
> Checkpoint doesn't request new buffers, but it does require the
> BufMgrLock in order to write all of the dirty buffers.
There is no BufMgrLock anymore in the clock-algorithm code, so I'm not
sure how much of this analysis is still relevant.
regards, tom lane
Simon Riggs <simon@2ndquadrant.com> writes: > Amdahl's Law tells me that looking at the checkpoints is the next best > action for tuning, since they add considerably to the average response > time. Looking at the oprofile for the run as a whole is missing out the > delayed transaction behaviour that occurs during checkpoints. Even aside from the effect it has on average response time. I would point out that many applications are governed by the worst case more than the average throughput. For a web server, for example (or any OLTP application in general), it doesn't matter if the database can handle x transactions/s on average. What matters is that 100% of the time the latency is below the actual rate of requests. If every 30m latency suddenly spikes up beyond that, even for only a minute, then it will fall behind in the requests. The user will effectively see a completely unresponsive web server. So I would really urge you to focus your attention on the maximum latency figure. It's at least if not *more* important than the average throughput number. PS That's why I was pushing before for the idea that the server should try to spread the I/O from one checkpoint out over more or less the time interval between checkpoints. If it's been 30m since the last checkpoint then you have about 30m to do the I/O for this checkpoint. (Though I would suggest a safety factor of aiming to be finished within 50% of the time.) -- greg
On Fri, 2005-03-04 at 20:10 -0500, Greg Stark wrote: > Simon Riggs <simon@2ndquadrant.com> writes: > > > Amdahl's Law tells me that looking at the checkpoints is the next best > > action for tuning, since they add considerably to the average response > > time. Looking at the oprofile for the run as a whole is missing out the > > delayed transaction behaviour that occurs during checkpoints. > > Even aside from the effect it has on average response time. I would point out > that many applications are governed by the worst case more than the average > throughput. > > For a web server, for example (or any OLTP application in general), it doesn't > matter if the database can handle x transactions/s on average. What matters is > that 100% of the time the latency is below the actual rate of requests. If > every 30m latency suddenly spikes up beyond that, even for only a minute, then > it will fall behind in the requests. The user will effectively see a > completely unresponsive web server. > > So I would really urge you to focus your attention on the maximum latency > figure. It's at least if not *more* important than the average throughput > number. Sorry Greg, clearly my English was poor. The checkpoints are the source of the peak latency on transactions, so we are in complete agreement. > PS That's why I was pushing before for the idea that the server should try to > spread the I/O from one checkpoint out over more or less the time interval > between checkpoints. If it's been 30m since the last checkpoint then you have > about 30m to do the I/O for this checkpoint. (Though I would suggest a safety > factor of aiming to be finished within 50% of the time.) I don't want to fix it before I know what the issue is. Best Regards, Simon Riggs
Simon Riggs <simon@2ndquadrant.com> writes: > The checkpoints are the source of the peak latency on transactions, so we > are in complete agreement. I realize that. I was just supporting your conclusion but saying that latency is important in its own right, not just as a means to achieving throughput. -- greg