Thread: 8.0.X and the ARC patent

8.0.X and the ARC patent

From
Bruce Momjian
Date:
FYI, core has discussed the pending IBM ARC patent and the usage of
those ideas in 8.0.

Tom has found a 2Q cache algorithm that predates the ARC patent and is
very similar to ARC.  The major difference is that it doesn't auto-size
the ARC sub-buffers.  

Core believes it is best to backpatch this 2Q algorithm into 8.0.X to
avoid any possible patent problems if the patent is granted and
enforced.

We are testing the use of the 2Q code to see if it has any performance
impact.  The 8.0.X release that uses 2Q will have more extensive testing
than a normal minor release.  8.1 will have a new cache algorithm that
hopefully will remove the buffer contention problems experienced by SMP
machines.

For development, this means we will _not_ have a shortened, non-initdb
8.1 release but a regular release cycle with the typical big batch of
features.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
pgsql@mohawksoft.com
Date:
Might it be possible to contact IBM directly and ask if they will allow
usage of the patent for PostgreSQL. They've let 500 patents for open
source, maybe they'll give a write off for this as well.

There is an advantage beyond just not having to re-write the code, but it
would also be sort of an IBM blessing, great PR.

I will be at "Linux World" and see if there is an IBM booth, maybe I can
get  some contact info.


> FYI, core has discussed the pending IBM ARC patent and the usage of
> those ideas in 8.0.
>
> Tom has found a 2Q cache algorithm that predates the ARC patent and is
> very similar to ARC.  The major difference is that it doesn't auto-size
> the ARC sub-buffers.
>
> Core believes it is best to backpatch this 2Q algorithm into 8.0.X to
> avoid any possible patent problems if the patent is granted and
> enforced.
>
> We are testing the use of the 2Q code to see if it has any performance
> impact.  The 8.0.X release that uses 2Q will have more extensive testing
> than a normal minor release.  8.1 will have a new cache algorithm that
> hopefully will remove the buffer contention problems experienced by SMP
> machines.
>
> For development, this means we will _not_ have a shortened, non-initdb
> 8.1 release but a regular release cycle with the typical big batch of
> features.
>
> --
>   Bruce Momjian                        |  http://candle.pha.pa.us
>   pgman@candle.pha.pa.us               |  (610) 359-1001
>   +  If your life is a hard drive,     |  13 Roberts Road
>   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> 19073
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq
>



Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
pgsql@mohawksoft.com wrote:
> Might it be possible to contact IBM directly and ask if they will allow
> usage of the patent for PostgreSQL. They've let 500 patents for open
> source, maybe they'll give a write off for this as well.
> 
> There is an advantage beyond just not having to re-write the code, but it
> would also be sort of an IBM blessing, great PR.
> 
> I will be at "Linux World" and see if there is an IBM booth, maybe I can
> get  some contact info.

I doubt they will give us something that extends to companies that sell
PostgreSQL so I don't see the point.

---------------------------------------------------------------------------


> 
> 
> > FYI, core has discussed the pending IBM ARC patent and the usage of
> > those ideas in 8.0.
> >
> > Tom has found a 2Q cache algorithm that predates the ARC patent and is
> > very similar to ARC.  The major difference is that it doesn't auto-size
> > the ARC sub-buffers.
> >
> > Core believes it is best to backpatch this 2Q algorithm into 8.0.X to
> > avoid any possible patent problems if the patent is granted and
> > enforced.
> >
> > We are testing the use of the 2Q code to see if it has any performance
> > impact.  The 8.0.X release that uses 2Q will have more extensive testing
> > than a normal minor release.  8.1 will have a new cache algorithm that
> > hopefully will remove the buffer contention problems experienced by SMP
> > machines.
> >
> > For development, this means we will _not_ have a shortened, non-initdb
> > 8.1 release but a regular release cycle with the typical big batch of
> > features.
> >
> > --
> >   Bruce Momjian                        |  http://candle.pha.pa.us
> >   pgman@candle.pha.pa.us               |  (610) 359-1001
> >   +  If your life is a hard drive,     |  13 Roberts Road
> >   +  Christ can be your backup.        |  Newtown Square, Pennsylvania
> > 19073
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> >                http://www.postgresql.org/docs/faq
> >
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings
> 

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
pgsql@mohawksoft.com
Date:
> pgsql@mohawksoft.com wrote:
>> Might it be possible to contact IBM directly and ask if they will allow
>> usage of the patent for PostgreSQL. They've let 500 patents for open
>> source, maybe they'll give a write off for this as well.
>>
>> There is an advantage beyond just not having to re-write the code, but
>> it
>> would also be sort of an IBM blessing, great PR.
>>
>> I will be at "Linux World" and see if there is an IBM booth, maybe I can
>> get  some contact info.
>
> I doubt they will give us something that extends to companies that sell
> PostgreSQL so I don't see the point.

Actually, I think that's wrong. IBM has been really gung-ho about Linux.
Of course this is an obvious movement against Microsoft domination of the
server market, but they have made some very strong open source statements
and have release about 500 patents to open source projects.

The current open source patents extend to companies that sell other
products, why not PostgreSQL as well?

There is a *LOT* of crap going on with patents, there are so many issues
and motives that is hard to keep track of why who is doing what. My bet is
that it is 50/50. It all depends if IBM wants to hurt Oracle more than it
wants to defned DB2.





Re: 8.0.X and the ARC patent

From
"Joshua D. Drake"
Date:
Bruce Momjian wrote:

>pgsql@mohawksoft.com wrote:
>
>
>>Might it be possible to contact IBM directly and ask if they will allow
>>usage of the patent for PostgreSQL. They've let 500 patents for open
>>source, maybe they'll give a write off for this as well.
>>
>>There is an advantage beyond just not having to re-write the code, but it
>>would also be sort of an IBM blessing, great PR.
>>
>>I will be at "Linux World" and see if there is an IBM booth, maybe I can
>>get  some contact info.
>>
>>
>
>I doubt they will give us something that extends to companies that sell
>PostgreSQL so I don't see the point.
>
>

Also if I recall didn't Tom already have a patch ready to be
tested for the q2 stuff?

Sincerely,

Joshua D. Drake

>---------------------------------------------------------------------------
>
>
>
>
>>
>>
>>>FYI, core has discussed the pending IBM ARC patent and the usage of
>>>those ideas in 8.0.
>>>
>>>Tom has found a 2Q cache algorithm that predates the ARC patent and is
>>>very similar to ARC.  The major difference is that it doesn't auto-size
>>>the ARC sub-buffers.
>>>
>>>Core believes it is best to backpatch this 2Q algorithm into 8.0.X to
>>>avoid any possible patent problems if the patent is granted and
>>>enforced.
>>>
>>>We are testing the use of the 2Q code to see if it has any performance
>>>impact.  The 8.0.X release that uses 2Q will have more extensive testing
>>>than a normal minor release.  8.1 will have a new cache algorithm that
>>>hopefully will remove the buffer contention problems experienced by SMP
>>>machines.
>>>
>>>For development, this means we will _not_ have a shortened, non-initdb
>>>8.1 release but a regular release cycle with the typical big batch of
>>>features.
>>>
>>>--
>>>  Bruce Momjian                        |  http://candle.pha.pa.us
>>>  pgman@candle.pha.pa.us               |  (610) 359-1001
>>>  +  If your life is a hard drive,     |  13 Roberts Road
>>>  +  Christ can be your backup.        |  Newtown Square, Pennsylvania
>>>19073
>>>
>>>---------------------------(end of broadcast)---------------------------
>>>TIP 5: Have you checked our extensive FAQ?
>>>
>>>               http://www.postgresql.org/docs/faq
>>>
>>>
>>>
>>---------------------------(end of broadcast)---------------------------
>>TIP 7: don't forget to increase your free space map settings
>>
>>
>>
>
>
>


--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd@commandprompt.com - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL


Attachment

Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Joshua D. Drake wrote:
> Bruce Momjian wrote:
> 
> >pgsql@mohawksoft.com wrote:
> >  
> >
> >>Might it be possible to contact IBM directly and ask if they will allow
> >>usage of the patent for PostgreSQL. They've let 500 patents for open
> >>source, maybe they'll give a write off for this as well.
> >>
> >>There is an advantage beyond just not having to re-write the code, but it
> >>would also be sort of an IBM blessing, great PR.
> >>
> >>I will be at "Linux World" and see if there is an IBM booth, maybe I can
> >>get  some contact info.
> >>    
> >>
> >
> >I doubt they will give us something that extends to companies that sell
> >PostgreSQL so I don't see the point.
> >  
> >
> 
> Also if I recall didn't Tom already have a patch ready to be
> tested for the q2 stuff?

Yes, he does.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
pgsql@mohawksoft.com writes:
> Might it be possible to contact IBM directly and ask if they will allow
> usage of the patent for PostgreSQL. They've let 500 patents for open
> source, maybe they'll give a write off for this as well.

If there were hard evidence that the ARC algorithm was far better than
the alternatives, it might be worth going in that direction.  But there
is no such evidence.  Jan has retracted his original opinion that the
ARC code is a big improvement over what we had before, and I haven't
seen anyone else putting up benchmark numbers that say we need to keep
ARC.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Tom Lane wrote:
> pgsql@mohawksoft.com writes:
> > Might it be possible to contact IBM directly and ask if they will allow
> > usage of the patent for PostgreSQL. They've let 500 patents for open
> > source, maybe they'll give a write off for this as well.
> 
> If there were hard evidence that the ARC algorithm was far better than
> the alternatives, it might be worth going in that direction.  But there
> is no such evidence.  Jan has retracted his original opinion that the
> ARC code is a big improvement over what we had before, and I haven't
> seen anyone else putting up benchmark numbers that say we need to keep
> ARC.

And ARC has locking requirements that will make it very hard to fix our
SMP buffer management problems in 8.1.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> And ARC has locking requirements that will make it very hard to fix our
> SMP buffer management problems in 8.1.

I am working on a buffer manager rewrite using the BufMgrLock breakup
and "clock sweep" management algorithm we've been discussing.  At the
moment it's passing the regression tests but I'm sure there's some bugs
left :-(.  I just now tried it on the infamous context-swap-storm test
case using a 4-way machine at Red Hat.  PG 8.0 shows 20K or more CS/sec
and under 30% CPU usage in this situation.  The patch shows 99% CPU
utilization and about 200 CS/sec (which is about nil, because the
machine shows ~100 CS/sec with nothing running except vmstat).

Still to be determined: what we lose in extra I/O from the presumably
less efficient cache management; also what sort of slowdown occurs on
a single-CPU machine that isn't going to get any benefit from the
increased amount of lock management.  But it looks promising.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Thomas Hallgren
Date:
pgsql@mohawksoft.com wrote:
>>pgsql@mohawksoft.com wrote:
>>
>>>Might it be possible to contact IBM directly and ask if they will allow
>>>usage of the patent for PostgreSQL. They've let 500 patents for open
>>>source, maybe they'll give a write off for this as well.
>>>
>>>There is an advantage beyond just not having to re-write the code, but
>>>it
>>>would also be sort of an IBM blessing, great PR.
>>>
>>>I will be at "Linux World" and see if there is an IBM booth, maybe I can
>>>get  some contact info.
>>
>>I doubt they will give us something that extends to companies that sell
>>PostgreSQL so I don't see the point.
> 
> 
> Actually, I think that's wrong. IBM has been really gung-ho about Linux.
> Of course this is an obvious movement against Microsoft domination of the
> server market, but they have made some very strong open source statements
> and have release about 500 patents to open source projects.
> 
> The current open source patents extend to companies that sell other
> products, why not PostgreSQL as well?
> 
> There is a *LOT* of crap going on with patents, there are so many issues
> and motives that is hard to keep track of why who is doing what. My bet is
> that it is 50/50. It all depends if IBM wants to hurt Oracle more than it
> wants to defned DB2.
> 
Seeing up close what IBM is doing with Eclipse, I must concur with this. 
I would be surprised if they said no. IBM knows that making revenue from 
software licenses is becoming increasingly harder. My impression is that 
they want to become less dependent on that and instead become more 
services oriented. This patent was filed before they started to change 
direction big time. Personally, I don't think they would care about it 
at all now.

So IMHO, ask IBM. It can't hurt.

Regards,
Thomas Hallgren



Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Tom Lane wrote:
> Still to be determined: what we lose in extra I/O from the presumably
> less efficient cache management; also what sort of slowdown occurs on
> a single-CPU machine that isn't going to get any benefit from the
> increased amount of lock management.  But it looks promising.

Yea, that was one of my questions --- the new buffer locking helps SMP,
but how much does it hurt single-cpu machines?  Do we need autodetection
or a GUC to control SMP-beneficial locking?

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> Still to be determined: what we lose in extra I/O from the presumably
>> less efficient cache management; also what sort of slowdown occurs on
>> a single-CPU machine that isn't going to get any benefit from the
>> increased amount of lock management.  But it looks promising.

> Yea, that was one of my questions --- the new buffer locking helps SMP,
> but how much does it hurt single-cpu machines?

So far I've not been able to measure any consistent difference, but you
know how much I trust pgbench ;-).  I hope that Mark Wong can give us
some results on the OSDL setup soon.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Simon Riggs
Date:
On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
> For development, this means we will _not_ have a shortened, non-initdb
> 8.1 release but a regular release cycle with the typical big batch of
> features.

Might we set a rough date for Beta freeze for 8.1 then?

September 30th 2005 ?

I see only benefit from publishing a not-before date now. It's up to
Core if it slips, but it'll really help with gaining funding if people
can accurately determine whether or not features can be added for
inclusion in the next release. There are lots of potential donors
waiting, so lets give them some certainty about which release their
payback will occur in....

Best Regards, Simon Riggs



Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Simon Riggs wrote:
> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
> > For development, this means we will _not_ have a shortened, non-initdb
> > 8.1 release but a regular release cycle with the typical big batch of
> > features.
> 
> Might we set a rough date for Beta freeze for 8.1 then?
> 
> September 30th 2005 ?
> 
> I see only benefit from publishing a not-before date now. It's up to
> Core if it slips, but it'll really help with gaining funding if people
> can accurately determine whether or not features can be added for
> inclusion in the next release. There are lots of potential donors
> waiting, so lets give them some certainty about which release their
> payback will occur in....

Yea, probably September, but you can't dump a huge feature on us in
August either without having talked about it first, so knowing the date
might not be that helpful.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
"Marc G. Fournier"
Date:
On Fri, 25 Feb 2005, Bruce Momjian wrote:

> Simon Riggs wrote:
>> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
>>> For development, this means we will _not_ have a shortened, non-initdb
>>> 8.1 release but a regular release cycle with the typical big batch of
>>> features.
>>
>> Might we set a rough date for Beta freeze for 8.1 then?
>>
>> September 30th 2005 ?
>>
>> I see only benefit from publishing a not-before date now. It's up to
>> Core if it slips, but it'll really help with gaining funding if people
>> can accurately determine whether or not features can be added for
>> inclusion in the next release. There are lots of potential donors
>> waiting, so lets give them some certainty about which release their
>> payback will occur in....
>
> Yea, probably September, but you can't dump a huge feature on us in
> August either without having talked about it first, so knowing the date
> might not be that helpful.

I thought we were looking at a 12-18 month cycle for 8.1?  Which would put 
beta around January '06, no?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: scrappy@hub.org           Yahoo!: yscrappy              ICQ: 7615664


Re: 8.0.X and the ARC patent

From
Andrew Dunstan
Date:

Bruce Momjian wrote:

>Simon Riggs wrote:
>  
>
>>On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
>>    
>>
>>>For development, this means we will _not_ have a shortened, non-initdb
>>>8.1 release but a regular release cycle with the typical big batch of
>>>features.
>>>      
>>>
>>Might we set a rough date for Beta freeze for 8.1 then?
>>
>>September 30th 2005 ?
>>
>>I see only benefit from publishing a not-before date now. It's up to
>>Core if it slips, but it'll really help with gaining funding if people
>>can accurately determine whether or not features can be added for
>>inclusion in the next release. There are lots of potential donors
>>waiting, so lets give them some certainty about which release their
>>payback will occur in....
>>    
>>
>
>Yea, probably September, but you can't dump a huge feature on us in
>August either without having talked about it first, so knowing the date
>might not be that helpful.
>  
>


Am I misunderstanding here? September 30th is 15 months after the 
feature freeze for 8.0. On that basis we might reasonably expect 8.1 to 
appear around April 2006. Is that what's intended?

cheers

andrew


Re: 8.0.X and the ARC patent

From
Simon Riggs
Date:
On Fri, 2005-02-25 at 00:18 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> > On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
> > > For development, this means we will _not_ have a shortened, non-initdb
> > > 8.1 release but a regular release cycle with the typical big batch of
> > > features.
> > 
> > Might we set a rough date for Beta freeze for 8.1 then?
> > 
> > September 30th 2005 ?
> > 
> > I see only benefit from publishing a not-before date now. It's up to
> > Core if it slips, but it'll really help with gaining funding if people
> > can accurately determine whether or not features can be added for
> > inclusion in the next release. There are lots of potential donors
> > waiting, so lets give them some certainty about which release their
> > payback will occur in....
> 
> Yea, probably September, but you can't dump a huge feature on us in
> August either without having talked about it first, so knowing the date
> might not be that helpful.
> 

[Sorry, just brought this up as part of another more general thread.]

That's fine. Just have two (rough dates):

- Major Feature Design Freeze - June 2005 (or 2Q2005)
Major features may require significant discussion, prototyping and
performance testing before agreement to include. You are advised that
major features presented for initial code review may not be accepted
after this date.

- Overall Feature Freeze - Sept 2005 (or 3Q2005)
All code implementing new features, large or small, should be complete
by this date. Features may be rejected if supporting tests and
documentation are not provided.

Best Regards, Simon Riggs



Re: 8.0.X and the ARC patent

From
Simon Riggs
Date:
On Fri, 2005-02-25 at 02:47 -0400, Marc G. Fournier wrote:
> On Fri, 25 Feb 2005, Bruce Momjian wrote:
> 
> > Simon Riggs wrote:
> >> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
> >>> For development, this means we will _not_ have a shortened, non-initdb
> >>> 8.1 release but a regular release cycle with the typical big batch of
> >>> features.
> >>
> >> Might we set a rough date for Beta freeze for 8.1 then?
> >>
> >> September 30th 2005 ?
> >>
> >> I see only benefit from publishing a not-before date now. It's up to
> >> Core if it slips, but it'll really help with gaining funding if people
> >> can accurately determine whether or not features can be added for
> >> inclusion in the next release. There are lots of potential donors
> >> waiting, so lets give them some certainty about which release their
> >> payback will occur in....
> >
> > Yea, probably September, but you can't dump a huge feature on us in
> > August either without having talked about it first, so knowing the date
> > might not be that helpful.
> 
> I thought we were looking at a 12-18 month cycle for 8.1?  Which would put 
> beta around January '06, no?

I'm happy with Core taking this decision, I'd just like to know +/- 1
month when that date is, i.e. which quarter of which year.

Best Regards, Simon Riggs



Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Marc G. Fournier wrote:
> On Fri, 25 Feb 2005, Bruce Momjian wrote:
> 
> > Simon Riggs wrote:
> >> On Mon, 2005-02-14 at 18:17 -0500, Bruce Momjian wrote:
> >>> For development, this means we will _not_ have a shortened, non-initdb
> >>> 8.1 release but a regular release cycle with the typical big batch of
> >>> features.
> >>
> >> Might we set a rough date for Beta freeze for 8.1 then?
> >>
> >> September 30th 2005 ?
> >>
> >> I see only benefit from publishing a not-before date now. It's up to
> >> Core if it slips, but it'll really help with gaining funding if people
> >> can accurately determine whether or not features can be added for
> >> inclusion in the next release. There are lots of potential donors
> >> waiting, so lets give them some certainty about which release their
> >> payback will occur in....
> >
> > Yea, probably September, but you can't dump a huge feature on us in
> > August either without having talked about it first, so knowing the date
> > might not be that helpful.
> 
> I thought we were looking at a 12-18 month cycle for 8.1?  Which would put 
> beta around January '06, no?

Oh, yea. I didn't realize people agreed with that, but I certainly
agree.

So that would put us at January.  Simon, I think the major issue is how
much does it help us to fix a date now vs. what is the benefit to
allowing us to state it later.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
"Marc G. Fournier" <scrappy@postgresql.org> writes:
>> Yea, probably September, but you can't dump a huge feature on us in
>> August either without having talked about it first, so knowing the date
>> might not be that helpful.

> I thought we were looking at a 12-18 month cycle for 8.1?  Which would put 
> beta around January '06, no?

Although we've dropped the idea of letting the ARC problem drive a very
short 8.1 cycle, I would still like to see us shoot for a relatively
short 8.1 cycle --- less than a year for sure.  The main reason is that
I think we'll be flushing out performance and feature issues in the
Windows port that we cannot reasonably back-patch into 8.0.*.  PITR also.
In general it seems to me that 8.1 will need to have a consolidation and
fill-in-the-blanks flavor after what we did for 8.0, and that will be
helped by a shorter devel cycle.

As a proposal: feature freeze maybe early summer (June or July), beta
maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with
a good tailwind).
        regards, tom lane


Re: 8.0.X and the ARC patent

From
"Matthew T. O'Connor"
Date:
Tom Lane wrote:

>Although we've dropped the idea of letting the ARC problem drive a very
>short 8.1 cycle, I would still like to see us shoot for a relatively
>short 8.1 cycle --- less than a year for sure.  The main reason is that
>I think we'll be flushing out performance and feature issues in the
>Windows port that we cannot reasonably back-patch into 8.0.*.  PITR also.
>In general it seems to me that 8.1 will need to have a consolidation and
>fill-in-the-blanks flavor after what we did for 8.0, and that will be
>helped by a shorter devel cycle.
>
>As a proposal: feature freeze maybe early summer (June or July), beta
>maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with
>a good tailwind).
>

That sounds good.  I would think that lots of users probably won't use 
the Windows port in production until 8.1 (performance reasons, paranoia 
etc...)  I would hate to make put such a long delay in their adoption 
plans. 

One thing to consider while discussing the length of the cycle is what 
features are people planning on putting in?  The 8.0 cycle had to be 
long due to the many huge improvements.  I'm not aware of any 8.1 plans 
that are that ambitious, so why plan a long cycle when there are no 
features requiring it?  Am I missing something?

Matthew



Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Tom Lane wrote:
> "Marc G. Fournier" <scrappy@postgresql.org> writes:
> >> Yea, probably September, but you can't dump a huge feature on us in
> >> August either without having talked about it first, so knowing the date
> >> might not be that helpful.
> 
> > I thought we were looking at a 12-18 month cycle for 8.1?  Which would put 
> > beta around January '06, no?
> 
> Although we've dropped the idea of letting the ARC problem drive a very
> short 8.1 cycle, I would still like to see us shoot for a relatively
> short 8.1 cycle --- less than a year for sure.  The main reason is that
> I think we'll be flushing out performance and feature issues in the
> Windows port that we cannot reasonably back-patch into 8.0.*.  PITR also.
> In general it seems to me that 8.1 will need to have a consolidation and
> fill-in-the-blanks flavor after what we did for 8.0, and that will be
> helped by a shorter devel cycle.
> 
> As a proposal: feature freeze maybe early summer (June or July), beta
> maybe Aug/Sep, final as always "when it's ready" (maybe Oct/Nov with
> a good tailwind).

That is fine with me too.  Let's see how mush 8.0 fixing we need in 8.1
and that will help determine the cutoff date, as will completed features
that we want to get into a public release.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
"Matthew T. O'Connor" <matthew@zeut.net> writes:
> One thing to consider while discussing the length of the cycle is what 
> features are people planning on putting in?  The 8.0 cycle had to be 
> long due to the many huge improvements.  I'm not aware of any 8.1 plans 
> that are that ambitious, so why plan a long cycle when there are no 
> features requiring it?  Am I missing something?

The subtext here is that people are trying to decide what they intend to
shoot for in this cycle, and so they are asking Core what the schedule
target is.  You really misunderstand the dynamics.  8.0 didn't start out
to be what it ended up being; there was no master plan, and never has
been.  The most organization we've ever had is for Core to set a feature
freeze target date well in advance.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
On Sat, Feb 19, 2005 at 12:57:52AM -0500, Tom Lane wrote:
> So far I've not been able to measure any consistent difference, but you
> know how much I trust pgbench ;-).  I hope that Mark Wong can give us
> some results on the OSDL setup soon.

Sorry for the delay, broken laptop, vacation, etc.

Here's a baseline against 8.0.1:       http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/       throughput
3639.97

Against CVS from 20050217 and the 2Q patch:       http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/
throughput3875.84
 

About a 6% increase, but if you look at the performance over time,
it's degrading steadily.  The latter throughput peaks near 5000.

Mark


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Mark Wong <markw@osdl.org> writes:
> About a 6% increase, but if you look at the performance over time,
> it's degrading steadily.  The latter throughput peaks near 5000.

Curious.  The immediate question is "does it ever flatten out, and
if so at what TPM rate compared to 8.0.1?"  Could you run the same
test for a longer duration?

Also, I suppose the large spike at 30 minutes is from a checkpoint?
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
> Mark Wong <markw@osdl.org> writes:
> > About a 6% increase, but if you look at the performance over time,
> > it's degrading steadily.  The latter throughput peaks near 5000.
> 
> Curious.  The immediate question is "does it ever flatten out, and
> if so at what TPM rate compared to 8.0.1?"  Could you run the same
> test for a longer duration?

The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
patch?  I can run a longer duration and see how it looks.

> Also, I suppose the large spike at 30 minutes is from a checkpoint?

Yes, checkpoint_timeout is set to 1800 seconds.

Mark


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Mark Wong <markw@osdl.org> writes:
> On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
>> Curious.  The immediate question is "does it ever flatten out, and
>> if so at what TPM rate compared to 8.0.1?"  Could you run the same
>> test for a longer duration?

> The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
> patch?  I can run a longer duration and see how it looks.

My point was that unpatched 8.0.1 seems to have a pretty level TPM
rate.  If the patched version levels out at something not far below
that, I'll be satisfied.  If it continues to degrade then I won't be
satisfied ... but the test stops short of telling what will happen.
If you could run it for 2 hours then we'd probably know enough.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote:
> Mark Wong <markw@osdl.org> writes:
> > On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
> >> Curious.  The immediate question is "does it ever flatten out, and
> >> if so at what TPM rate compared to 8.0.1?"  Could you run the same
> >> test for a longer duration?
> 
> > The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
> > patch?  I can run a longer duration and see how it looks.
> 
> My point was that unpatched 8.0.1 seems to have a pretty level TPM
> rate.  If the patched version levels out at something not far below
> that, I'll be satisfied.  If it continues to degrade then I won't be
> satisfied ... but the test stops short of telling what will happen.
> If you could run it for 2 hours then we'd probably know enough.

Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/

I ran it for 3 hours, just in case, and the charts suggest it flattens
out after 2 hours.

Mark


Re: 8.0.X and the ARC patent

From
Dave Cramer
Date:
I was just looking at the config parameters, and you have the shared 
buffers set to 60k, and the effective cache set to 1k ????

Dave

Mark Wong wrote:

>On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote:
>  
>
>>Mark Wong <markw@osdl.org> writes:
>>    
>>
>>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
>>>      
>>>
>>>>Curious.  The immediate question is "does it ever flatten out, and
>>>>if so at what TPM rate compared to 8.0.1?"  Could you run the same
>>>>test for a longer duration?
>>>>        
>>>>
>>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
>>>patch?  I can run a longer duration and see how it looks.
>>>      
>>>
>>My point was that unpatched 8.0.1 seems to have a pretty level TPM
>>rate.  If the patched version levels out at something not far below
>>that, I'll be satisfied.  If it continues to degrade then I won't be
>>satisfied ... but the test stops short of telling what will happen.
>>If you could run it for 2 hours then we'd probably know enough.
>>    
>>
>
>Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
>    http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
>
>I ran it for 3 hours, just in case, and the charts suggest it flattens
>out after 2 hours.
>
>Mark
>
>---------------------------(end of broadcast)---------------------------
>TIP 7: don't forget to increase your free space map settings
>
>
>  
>

-- 
Dave Cramer
http://www.postgresintl.com
519 939 0336
ICQ#14675561



Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
Yes, those parameters are based on a series of test results
here:http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html

Run 264 provided the best results, so I'm trying to continue with the
database parameters used there.

Mark

On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote:
> I was just looking at the config parameters, and you have the shared 
> buffers set to 60k, and the effective cache set to 1k ????
> 
> Dave
> 
> Mark Wong wrote:
> 
> >On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote:
> > 
> >
> >>Mark Wong <markw@osdl.org> writes:
> >>   
> >>
> >>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
> >>>     
> >>>
> >>>>Curious.  The immediate question is "does it ever flatten out, and
> >>>>if so at what TPM rate compared to 8.0.1?"  Could you run the same
> >>>>test for a longer duration?
> >>>>       
> >>>>
> >>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
> >>>patch?  I can run a longer duration and see how it looks.
> >>>     
> >>>
> >>My point was that unpatched 8.0.1 seems to have a pretty level TPM
> >>rate.  If the patched version levels out at something not far below
> >>that, I'll be satisfied.  If it continues to degrade then I won't be
> >>satisfied ... but the test stops short of telling what will happen.
> >>If you could run it for 2 hours then we'd probably know enough.
> >>   
> >>
> >
> >Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
> >    http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
> >
> >I ran it for 3 hours, just in case, and the charts suggest it flattens
> >out after 2 hours.
> >
> >Mark


Re: 8.0.X and the ARC patent

From
Dave Cramer
Date:
OK. I doubt that it impacts the results of the particular test, but it 
is non-intuitive (in my mind at least)
Did you change anything else between 263 and 264? From the table it 
appears that you are changing vm parameters
as well as database configuration parameters between runs ?

Dave

Mark Wong wrote:

>Yes, those parameters are based on a series of test results here:
>    http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html
>
>Run 264 provided the best results, so I'm trying to continue with the
>database parameters used there.
>
>Mark
>
>On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote:
>  
>
>>I was just looking at the config parameters, and you have the shared 
>>buffers set to 60k, and the effective cache set to 1k ????
>>
>>Dave
>>
>>Mark Wong wrote:
>>
>>    
>>
>>>On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote:
>>>
>>>
>>>      
>>>
>>>>Mark Wong <markw@osdl.org> writes:
>>>>  
>>>>
>>>>        
>>>>
>>>>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>>>>Curious.  The immediate question is "does it ever flatten out, and
>>>>>>if so at what TPM rate compared to 8.0.1?"  Could you run the same
>>>>>>test for a longer duration?
>>>>>>      
>>>>>>
>>>>>>            
>>>>>>
>>>>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
>>>>>patch?  I can run a longer duration and see how it looks.
>>>>>    
>>>>>
>>>>>          
>>>>>
>>>>My point was that unpatched 8.0.1 seems to have a pretty level TPM
>>>>rate.  If the patched version levels out at something not far below
>>>>that, I'll be satisfied.  If it continues to degrade then I won't be
>>>>satisfied ... but the test stops short of telling what will happen.
>>>>If you could run it for 2 hours then we'd probably know enough.
>>>>  
>>>>
>>>>        
>>>>
>>>Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
>>>    http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
>>>
>>>I ran it for 3 hours, just in case, and the charts suggest it flattens
>>>out after 2 hours.
>>>
>>>Mark
>>>      
>>>
>
>
>  
>

-- 
Dave Cramer
http://www.postgresintl.com
519 939 0336
ICQ#14675561



Re: 8.0.X and the ARC patent

From
Michael Adler
Date:
On Wed, Mar 02, 2005 at 07:21:41AM -0800, Mark Wong wrote:
> Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
>     http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
> 
> I ran it for 3 hours, just in case, and the charts suggest it flattens
> out after 2 hours.

Looking at the "Response Time Charts" 

8.0.1/ARC
http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html

20050301 with 2Q patch
http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html

It seems like the average response time has gone down, but the worse
case ceiling has raised about 35%.
-Mike


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Mark Wong <markw@osdl.org> writes:
> Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
>     http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/

> I ran it for 3 hours, just in case, and the charts suggest it flattens
> out after 2 hours.

Thanks.  This seems odd though, since it appears to level out at
something above 4K TPM.  Your previous run
http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/
shows it dropping to 3500 or so.  What changed?
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote:
> Mark Wong <markw@osdl.org> writes:
> > Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
> >     http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
> 
> > I ran it for 3 hours, just in case, and the charts suggest it flattens
> > out after 2 hours.
> 
> Thanks.  This seems odd though, since it appears to level out at
> something above 4K TPM.  Your previous run
> http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/
> shows it dropping to 3500 or so.  What changed?

Other than pulling from CVS at a different time, it should all be
the same parameters, etc.

Mark


Re: 8.0.X and the ARC patent

From
Mark Wong
Date:
Between those two runs, the vm parameters are the same.

Mark

On Wed, Mar 02, 2005 at 11:04:21AM -0500, Dave Cramer wrote:
> OK. I doubt that it impacts the results of the particular test, but it 
> is non-intuitive (in my mind at least)
> Did you change anything else between 263 and 264? From the table it 
> appears that you are changing vm parameters
> as well as database configuration parameters between runs ?
> 
> Dave
> 
> Mark Wong wrote:
> 
> >Yes, those parameters are based on a series of test results here:
> >    http://www.osdl.org/projects/dbt2dev/results/pgsql/rc4.html
> >
> >Run 264 provided the best results, so I'm trying to continue with the
> >database parameters used there.
> >
> >Mark
> >
> >On Wed, Mar 02, 2005 at 10:41:57AM -0500, Dave Cramer wrote:
> > 
> >
> >>I was just looking at the config parameters, and you have the shared 
> >>buffers set to 60k, and the effective cache set to 1k ????
> >>
> >>Dave
> >>
> >>Mark Wong wrote:
> >>
> >>   
> >>
> >>>On Tue, Mar 01, 2005 at 05:17:07PM -0500, Tom Lane wrote:
> >>>
> >>>
> >>>     
> >>>
> >>>>Mark Wong <markw@osdl.org> writes:
> >>>> 
> >>>>
> >>>>       
> >>>>
> >>>>>On Tue, Mar 01, 2005 at 04:57:11PM -0500, Tom Lane wrote:
> >>>>>   
> >>>>>
> >>>>>         
> >>>>>
> >>>>>>Curious.  The immediate question is "does it ever flatten out, and
> >>>>>>if so at what TPM rate compared to 8.0.1?"  Could you run the same
> >>>>>>test for a longer duration?
> >>>>>>     
> >>>>>>
> >>>>>>           
> >>>>>>
> >>>>>The comparison was against 8.0.1, or did you mean 8.0.1 with the 2Q
> >>>>>patch?  I can run a longer duration and see how it looks.
> >>>>>   
> >>>>>
> >>>>>         
> >>>>>
> >>>>My point was that unpatched 8.0.1 seems to have a pretty level TPM
> >>>>rate.  If the patched version levels out at something not far below
> >>>>that, I'll be satisfied.  If it continues to degrade then I won't be
> >>>>satisfied ... but the test stops short of telling what will happen.
> >>>>If you could run it for 2 hours then we'd probably know enough.
> >>>> 
> >>>>
> >>>>       
> >>>>
> >>>Ah, ok.  I've reapplied the 2Q patch to CVS from 20050301:
> >>>    http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/
> >>>
> >>>I ran it for 3 hours, just in case, and the charts suggest it flattens
> >>>out after 2 hours.
> >>>
> >>>Mark
> >>>     
> >>>
> >
> >
> > 
> >
> 
> -- 
> Dave Cramer
> http://www.postgresintl.com
> 519 939 0336
> ICQ#14675561

-- 
Mark Wong - - markw@osdl.org
Open Source Development Lab Inc - A non-profit corporation
12725 SW Millikan Way - Suite 400 - Beaverton, OR 97005
(503) 626-2455 (office)
(503) 626-2436 (fax)
http://developer.osdl.org/markw/


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Mark Wong <markw@osdl.org> writes:
> On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote:
>> Thanks.  This seems odd though, since it appears to level out at
>> something above 4K TPM.  Your previous run
>> http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/
>> shows it dropping to 3500 or so.  What changed?

> Other than pulling from CVS at a different time, it should all be
> the same parameters, etc.

Hmph.  The truth is probably somewhere in between these two curves.
But in any case, I think we can make the conclusion we wanted to
make: 2Q isn't seriously worse than ARC.  Since this is a dead line
of development anyway in view of the early results with the clock
sweep algorithm, I don't think there's any need to spend more time
measuring the differences carefully.

I'll go ahead and apply the 2Q patch to the 8.0 branch, unless there
are objections?
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Michael Adler <adler@pobox.com> writes:
> Looking at the "Response Time Charts" 

> 8.0.1/ARC
> http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html

> 20050301 with 2Q patch
> http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html

> It seems like the average response time has gone down, but the worse
> case ceiling has raised about 35%.

The worst cases are associated with checkpoints.  I'm not sure why a
checkpoint would have a greater effect on the 2Q system than an ARC
system --- checkpoint doesn't request any new buffers so you'd think
it'd be independent.  Maybe this says that the bgwriter is less
effective with 2Q, so that there are more dirty buffers remaining to
be written at the checkpoint?  But why?
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Bruce Momjian
Date:
Tom Lane wrote:
> Mark Wong <markw@osdl.org> writes:
> > On Wed, Mar 02, 2005 at 03:15:54PM -0500, Tom Lane wrote:
> >> Thanks.  This seems odd though, since it appears to level out at
> >> something above 4K TPM.  Your previous run
> >> http://www.osdl.org/projects/dbt2dev/results/dev4-010/311/
> >> shows it dropping to 3500 or so.  What changed?
> 
> > Other than pulling from CVS at a different time, it should all be
> > the same parameters, etc.
> 
> Hmph.  The truth is probably somewhere in between these two curves.
> But in any case, I think we can make the conclusion we wanted to
> make: 2Q isn't seriously worse than ARC.  Since this is a dead line
> of development anyway in view of the early results with the clock
> sweep algorithm, I don't think there's any need to spend more time
> measuring the differences carefully.

He reported a huge benefit in current CVS, like 30% --- was that because
of the clock algorithm?

> I'll go ahead and apply the 2Q patch to the 8.0 branch, unless there
> are objections?

Good.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
359-1001+  If your life is a hard drive,     |  13 Roberts Road +  Christ can be your backup.        |  Newtown Square,
Pennsylvania19073
 


Re: 8.0.X and the ARC patent

From
Josh Berkus
Date:
Tom,

I should be able to run more OLTP benchmarks, and a DSS benchmark, within the 
next week.   Please wait until I complete those before considering an 8.0.2 
release with the new code.    The machine and setup Mark is testing on is 
kind of end-of-the-curve; I have some more middle-of-the-road machines to 
test.

I also want to run one 6-hour or 12-hour test to see if the flattening out is 
actually flat.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco


Re: 8.0.X and the ARC patent

From
Greg Stark
Date:
Dave Cramer <pg@fastcrypt.com> writes:

> I was just looking at the config parameters, and you have the shared buffers
> set to 60k, and the effective cache set to 1k ????

I was actually going to suggest that the performance degradation might be
because of an excessively high shared_buffers setting. That was before I saw
this comment.

The only reason I could imagine the performance degradation would be because
more and more CPU time is being spent traversing the 2Q LRU buffer lists.

I would try it with a shared buffer setting of 10k to see if it levels out
sooner at a higher TPM.

I would also suggest setting checkpoint_timeout to something more realistic.
All your 60m tests that show a single checkpoint in the middle are being
deceptive since half the data in the test hasn't even been checkpointed. You
should have enough checkpoints in your test that they're represented in the
results realistically.

If you want 60m to be a reasonably representative sample then I would suggest
a checkpoint_timeout of 300-600 (ie, checkpoints every 5-10m) so you get 10-20
checkpoints in the result. And so that a maximum of 5-10% of the data isn't
being checkpointed in the test.

That would also make those huge performance dropouts a little less dramatic.
And it might give us a chance to see how effective the bgwriter is at
smoothing them out. Personally, as a user, I think it's more important to look
at the maximum transaction latency than the average throughput.

-- 
greg



Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> I should be able to run more OLTP benchmarks, and a DSS benchmark, within the 
> next week.   Please wait until I complete those before considering an 8.0.2 
> release with the new code.

Sure, there's no hurry to push out 8.0.2 (and we need to have some beta
testing done on it anyway).  I have committed the proposed patch into
the REL8_0_STABLE branch, so you can just pull the branch tip from CVS
to do testing.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Simon Riggs
Date:
On Wed, 2005-03-02 at 20:55 -0500, Tom Lane wrote:
> Michael Adler <adler@pobox.com> writes:
> > Looking at the "Response Time Charts" 
> 
> > 8.0.1/ARC
> > http://www.osdl.org/projects/dbt2dev/results/dev4-010/309/rt.html
> 
> > 20050301 with 2Q patch
> > http://www.osdl.org/projects/dbt2dev/results/dev4-010/313/rt.html
> 
> > It seems like the average response time has gone down, but the worse
> > case ceiling has raised about 35%.
> 
> The worst cases are associated with checkpoints.  I'm not sure why a
> checkpoint would have a greater effect on the 2Q system than an ARC
> system --- checkpoint doesn't request any new buffers so you'd think
> it'd be independent.  Maybe this says that the bgwriter is less
> effective with 2Q, so that there are more dirty buffers remaining to
> be written at the checkpoint?  But why?

The pattern seems familiar. Reduced average response time increases
total throughput, which on this test means we have more dirty buffers to
write at checkpoint time.

I would not neccessarily suspect 2Q over ARC, at least initially.

The pattern of behaviour is similar across ARC, 2Q and Clock, though the
checkpoint points differ in intensity. The latter makes me suspect
BufMgrLock contention or similar.

There is a two-level effect at Checkpoint time...first we have the write
from PostgreSQL buffers to OS cache, then we have the write from OS
cache to disk by the pdflush daemon. At this point, I'm not certain
whether the delay is caused by the checkpointing or the pdflush daemons.
Mark and I had discussed some investigations around that. This behaviour
is new in the 2.6 kernel, so it is possible there is an unpleasant
interaction there, though I do not wish to cast random blame.

Checkpoint doesn't request new buffers, but it does require the
BufMgrLock in order to write all of the dirty buffers. It could be that
the I/Os map direct to OS cache, so that the tight loop to write out
dirty buffers causes such an extreme backlog for the BufMgrLock that it
takes more than a minute to clear and return to normal contention.

It could be that at checkpoint time, the number of writes exceeds the
dirty_ratio and the kernel forces the checkpoint process to bypass the
cache and pdflush daemons altogether, and performing the I/O itself.
Single-threaded, this would display the scalability profile we see. Some
kernel level questions in there...

There is no documented event-state model for LWlock acquisition, so it
might be possible that there is a complex bottleneck in amongst them.

Amdahl's Law tells me that looking at the checkpoints is the next best
action for tuning, since they add considerably to the average response
time. Looking at the oprofile for the run as a whole is missing out the
delayed transaction behaviour that occurs during checkpoints.

I would like to try and catch an oprofile of the system while performing
a checkpoint, as a way to give us some clues. Perhaps that could be
achieved by forcing a manual checkpoint as superuser, and making that
interaction cause a switch to a new oprofile output file.

Best Regards, Simon Riggs





Re: 8.0.X and the ARC patent

From
Tom Lane
Date:
Simon Riggs <simon@2ndquadrant.com> writes:
> Checkpoint doesn't request new buffers, but it does require the
> BufMgrLock in order to write all of the dirty buffers.

There is no BufMgrLock anymore in the clock-algorithm code, so I'm not
sure how much of this analysis is still relevant.
        regards, tom lane


Re: 8.0.X and the ARC patent

From
Greg Stark
Date:
Simon Riggs <simon@2ndquadrant.com> writes:

> Amdahl's Law tells me that looking at the checkpoints is the next best
> action for tuning, since they add considerably to the average response
> time. Looking at the oprofile for the run as a whole is missing out the
> delayed transaction behaviour that occurs during checkpoints.

Even aside from the effect it has on average response time. I would point out
that many applications are governed by the worst case more than the average
throughput.

For a web server, for example (or any OLTP application in general), it doesn't
matter if the database can handle x transactions/s on average. What matters is
that 100% of the time the latency is below the actual rate of requests. If
every 30m latency suddenly spikes up beyond that, even for only a minute, then
it will fall behind in the requests. The user will effectively see a
completely unresponsive web server.

So I would really urge you to focus your attention on the maximum latency
figure. It's at least if not *more* important than the average throughput
number.



PS That's why I was pushing before for the idea that the server should try to
spread the I/O from one checkpoint out over more or less the time interval
between checkpoints. If it's been 30m since the last checkpoint then you have
about 30m to do the I/O for this checkpoint. (Though I would suggest a safety
factor of aiming to be finished within 50% of the time.)

-- 
greg



Re: 8.0.X and the ARC patent

From
Simon Riggs
Date:
On Fri, 2005-03-04 at 20:10 -0500, Greg Stark wrote:
> Simon Riggs <simon@2ndquadrant.com> writes:
> 
> > Amdahl's Law tells me that looking at the checkpoints is the next best
> > action for tuning, since they add considerably to the average response
> > time. Looking at the oprofile for the run as a whole is missing out the
> > delayed transaction behaviour that occurs during checkpoints.
> 
> Even aside from the effect it has on average response time. I would point out
> that many applications are governed by the worst case more than the average
> throughput.
> 
> For a web server, for example (or any OLTP application in general), it doesn't
> matter if the database can handle x transactions/s on average. What matters is
> that 100% of the time the latency is below the actual rate of requests. If
> every 30m latency suddenly spikes up beyond that, even for only a minute, then
> it will fall behind in the requests. The user will effectively see a
> completely unresponsive web server.
> 
> So I would really urge you to focus your attention on the maximum latency
> figure. It's at least if not *more* important than the average throughput
> number.

Sorry Greg, clearly my English was poor. 

The checkpoints are the source of the peak latency on transactions, so
we are in complete agreement. 

> PS That's why I was pushing before for the idea that the server should try to
> spread the I/O from one checkpoint out over more or less the time interval
> between checkpoints. If it's been 30m since the last checkpoint then you have
> about 30m to do the I/O for this checkpoint. (Though I would suggest a safety
> factor of aiming to be finished within 50% of the time.)

I don't want to fix it before I know what the issue is.

Best Regards, Simon Riggs



Re: 8.0.X and the ARC patent

From
Greg Stark
Date:
Simon Riggs <simon@2ndquadrant.com> writes:

> The checkpoints are the source of the peak latency on transactions, so we
> are in complete agreement.

I realize that. I was just supporting your conclusion but saying that latency
is important in its own right, not just as a means to achieving throughput.

-- 
greg