Thread: Wierd performance issue with 8.1cvs

Wierd performance issue with 8.1cvs

From
Josh Berkus
Date:
Folks,

At OSDL we're seeing a wierd performance crash on 8.1cvs.   What's wierd about 
it is that it doesn't happen all the time -- about 1 out of 4 test runs.    
What it looks like happens sometimes is that performance drops dramatically 
at the first checkpoint, and never comes back.  But there's oprofiles and 
things to make a more insightful analysis:

3 test runs exhibit it:
http://khack.osdl.org/stp/301531/0.html
http://khack.osdl.org/stp/301736/0.html
http://khack.osdl.org/stp/301730/0.html

Note that these are all different day's builds.

-- 
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco


Re: Wierd performance issue with 8.1cvs

From
Alvaro Herrera
Date:
On Wed, Apr 20, 2005 at 11:09:19AM -0700, Josh Berkus wrote:

> At OSDL we're seeing a wierd performance crash on 8.1cvs.   What's wierd about 
> it is that it doesn't happen all the time -- about 1 out of 4 test runs.    
> What it looks like happens sometimes is that performance drops dramatically 
> at the first checkpoint, and never comes back.  But there's oprofiles and 
> things to make a more insightful analysis:

What is the "wait" line in the Processor Utilization graphic?  Does it
have anything to do with spinlocks?  Was it so high in 8.0 tests?

-- 
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
Jude: I wish humans laid eggs
Ringlord: Why would you want humans to lay eggs?
Jude: So I can eat them


Re: Wierd performance issue with 8.1cvs

From
Greg Stark
Date:
Josh Berkus <josh@agliodbs.com> writes:

> Folks,
> 
> At OSDL we're seeing a wierd performance crash on 8.1cvs.   What's wierd about 
> it is that it doesn't happen all the time -- about 1 out of 4 test runs.    
> What it looks like happens sometimes is that performance drops dramatically 
> at the first checkpoint, and never comes back.  But there's oprofiles and 
> things to make a more insightful analysis:
> 
> 3 test runs exhibit it:
> http://khack.osdl.org/stp/301531/0.html
> http://khack.osdl.org/stp/301736/0.html
> http://khack.osdl.org/stp/301730/0.html

That dropoff at 60 minutes is the *first* checkpoint?! On an 80m test run?

That's a totally unrealistic configuration. Do you have any reason to think
the drop-off isn't just because all that pending i/o that you've postponed for
so long is finally having to get written out? Worse, it's forcing Postgres to
fsync files after 60m of i/o has been performed, flushing huge queues of i/o.

The benchmarks performed in this configuration are completely bogus. They
aren't including the time to checkpoint the last 20m of i/o, a quarter of all
the i/o in the test.

You really have to lower the checkpoint timeout to something realistic, like
5m or so. Otherwise these tests are just useless.


-- 
greg



Re: Wierd performance issue with 8.1cvs

From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> At OSDL we're seeing a wierd performance crash on 8.1cvs.

Permit me a digression for a pet peeve...

"Weird" is spelled "weird".  Not "wierd".

Yes, I know the nursery rhyme as well as you do --- i before e except
after c, etc etc.  But weird is spelled weirdly.  Appropriate, isn't it?
        regards, tom lane

PS: If you need an authoritative reference, will the OED do?
http://www.askoxford.com/concise_oed/weird?view=uk


Re: Wierd performance issue with 8.1cvs

From
Christopher Kings-Lynne
Date:
> Permit me a digression for a pet peeve...
> 
> "Weird" is spelled "weird".  Not "wierd".
> 
> Yes, I know the nursery rhyme as well as you do --- i before e except
> after c, etc etc.  But weird is spelled weirdly.  Appropriate, isn't it?

It's also my pet peeve, but I long ago stopped bothering to correct people.

One may also wish to consider "feisty" :)

Chris


Re: Wierd performance issue with 8.1cvs

From
Oleg Bartunov
Date:
On Sun, 24 Apr 2005, Christopher Kings-Lynne wrote:

>> Permit me a digression for a pet peeve...
>> 
>> "Weird" is spelled "weird".  Not "wierd".
>> 
>> Yes, I know the nursery rhyme as well as you do --- i before e except
>> after c, etc etc.  But weird is spelled weirdly.  Appropriate, isn't it?
>
> It's also my pet peeve, but I long ago stopped bothering to correct people.

I think it's quite useful to correct, because many of us use  english only
when reading mailing lists and documentation :)

    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Re: Wierd performance issue with 8.1cvs

From
Thomas Hallgren
Date:
Oleg Bartunov wrote:
> I think it's quite useful to correct, because many of us use  english only
> when reading mailing lists and documentation :)
> 
I think that it's important to refrain from corrections on a public 
forum as long as the essence of the message is clear. Some people might 
get offended, others might just stop writing because they get 
intimidated by the seemingly high demands on correct spelling or 
sentence structure.

I appreciate getting corrected by people I know in a limited forum. I 
would not expect it when I do a mistakes here. Can't say it ever has 
happend although there's often good grounds for it so I have nothing to 
complain about. :-)

Regards,
Thomas Hallgren



Re: Wierd performance issue with 8.1cvs

From
"John Hansen"
Date:
> I appreciate getting corrected by people I know in a limited
> forum. I would not expect it when I do a mistakes here. Can't
> say it ever has happend although there's often good grounds
> for it so I have nothing to complain about. :-)

I think you meant to say 'I Can't say it ever has happened...'

:)

... John


Re: Wierd performance issue with 8.1cvs

From
Oleg Bartunov
Date:
On Sun, 24 Apr 2005, Thomas Hallgren wrote:

> Oleg Bartunov wrote:
>> I think it's quite useful to correct, because many of us use  english only
>> when reading mailing lists and documentation :)
>> 
> I think that it's important to refrain from corrections on a public forum as 
> long as the essence of the message is clear. Some people might get offended, 
> others might just stop writing because they get intimidated by the seemingly 
> high demands on correct spelling or sentence structure.
>
> I appreciate getting corrected by people I know in a limited forum. I would 
> not expect it when I do a mistakes here. Can't say it ever has happend 
> although there's often good grounds for it so I have nothing to complain 
> about. :-)

It's difficult to opposed you but "limited forum" is not we could afford.
When I see a lot of mistakes, crude words in russian forums I always feel
myself uncomfortable. Unfortunately, I can't imagine how it looks for
english speaking people. Of course, there are simple misprints which could
be painlessly ommited, but I see nothing offending against correcting
gross mistakes. Usually, people just too busy to notice them.

>
> Regards,
> Thomas Hallgren
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
    Regards,        Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Re: Wierd performance issue with 8.1cvs

From
Thomas Hallgren
Date:
Oleg Bartunov wrote:

> On Sun, 24 Apr 2005, Thomas Hallgren wrote:
>
>> Oleg Bartunov wrote:
>>
>>> I think it's quite useful to correct, because many of us use  
>>> english only
>>> when reading mailing lists and documentation :)
>>>
>> I think that it's important to refrain from corrections on a public 
>> forum as long as the essence of the message is clear. Some people 
>> might get offended, others might just stop writing because they get 
>> intimidated by the seemingly high demands on correct spelling or 
>> sentence structure.
>>
>> I appreciate getting corrected by people I know in a limited forum. I 
>> would not expect it when I do a mistakes here. Can't say it ever has 
>> happend although there's often good grounds for it so I have nothing 
>> to complain about. :-)
>
>
> It's difficult to opposed you but "limited forum" is not we could afford.
> When I see a lot of mistakes, crude words in russian forums I always feel
> myself uncomfortable. Unfortunately, I can't imagine how it looks for
> english speaking people. Of course, there are simple misprints which 
> could
> be painlessly ommited, but I see nothing offending against correcting
> gross mistakes. Usually, people just too busy to notice them.

This Sentience has tree errors.

Regards,
Thomas Hallgren



Re: W[i/e]rd performance issue with 8.1cvs

From
Josh Berkus
Date:
Guys,

> >> "Weird" is spelled "weird".  Not "wierd".

OK, spelling errors taken into account.  Now could we perhaps address the 
**postgresql** errors?

I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs.   
It's obviously a serious stability issue, whatever is causing it.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco


Re: W[i/e]rd performance issue with 8.1cvs

From
Tom Lane
Date:
Josh Berkus <josh@agliodbs.com> writes:
> I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs.   
> It's obviously a serious stability issue, whatever is causing it.

I concur with the upthread suggestion that it may come from not doing
checkpoints in a realistic fashion, thereby allowing too much queued
I/O work to build up.  What happens if you set the checkpoint interval
to 5 or 10 minutes?
        regards, tom lane


Re: W[i/e]rd performance issue with 8.1cvs

From
Josh Berkus
Date:
Tom,

> I concur with the upthread suggestion that it may come from not doing
> checkpoints in a realistic fashion, thereby allowing too much queued
> I/O work to build up.  What happens if you set the checkpoint interval
> to 5 or 10 minutes?

I'll test.  Keep in mind that it takes me a couple of days to run a new test
(I've got 11 in the queue right now) so it'll be a bit.

--
Josh Berkus
Aglio Database Solutions
San Francisco