Thread: Wierd performance issue with 8.1cvs
Folks, At OSDL we're seeing a wierd performance crash on 8.1cvs. What's wierd about it is that it doesn't happen all the time -- about 1 out of 4 test runs. What it looks like happens sometimes is that performance drops dramatically at the first checkpoint, and never comes back. But there's oprofiles and things to make a more insightful analysis: 3 test runs exhibit it: http://khack.osdl.org/stp/301531/0.html http://khack.osdl.org/stp/301736/0.html http://khack.osdl.org/stp/301730/0.html Note that these are all different day's builds. -- --Josh Josh Berkus Aglio Database Solutions San Francisco
On Wed, Apr 20, 2005 at 11:09:19AM -0700, Josh Berkus wrote: > At OSDL we're seeing a wierd performance crash on 8.1cvs. What's wierd about > it is that it doesn't happen all the time -- about 1 out of 4 test runs. > What it looks like happens sometimes is that performance drops dramatically > at the first checkpoint, and never comes back. But there's oprofiles and > things to make a more insightful analysis: What is the "wait" line in the Processor Utilization graphic? Does it have anything to do with spinlocks? Was it so high in 8.0 tests? -- Alvaro Herrera (<alvherre[@]dcc.uchile.cl>) Jude: I wish humans laid eggs Ringlord: Why would you want humans to lay eggs? Jude: So I can eat them
Josh Berkus <josh@agliodbs.com> writes: > Folks, > > At OSDL we're seeing a wierd performance crash on 8.1cvs. What's wierd about > it is that it doesn't happen all the time -- about 1 out of 4 test runs. > What it looks like happens sometimes is that performance drops dramatically > at the first checkpoint, and never comes back. But there's oprofiles and > things to make a more insightful analysis: > > 3 test runs exhibit it: > http://khack.osdl.org/stp/301531/0.html > http://khack.osdl.org/stp/301736/0.html > http://khack.osdl.org/stp/301730/0.html That dropoff at 60 minutes is the *first* checkpoint?! On an 80m test run? That's a totally unrealistic configuration. Do you have any reason to think the drop-off isn't just because all that pending i/o that you've postponed for so long is finally having to get written out? Worse, it's forcing Postgres to fsync files after 60m of i/o has been performed, flushing huge queues of i/o. The benchmarks performed in this configuration are completely bogus. They aren't including the time to checkpoint the last 20m of i/o, a quarter of all the i/o in the test. You really have to lower the checkpoint timeout to something realistic, like 5m or so. Otherwise these tests are just useless. -- greg
Josh Berkus <josh@agliodbs.com> writes:
> At OSDL we're seeing a wierd performance crash on 8.1cvs.
Permit me a digression for a pet peeve...
"Weird" is spelled "weird". Not "wierd".
Yes, I know the nursery rhyme as well as you do --- i before e except
after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it?
regards, tom lane
PS: If you need an authoritative reference, will the OED do?
http://www.askoxford.com/concise_oed/weird?view=uk
> Permit me a digression for a pet peeve... > > "Weird" is spelled "weird". Not "wierd". > > Yes, I know the nursery rhyme as well as you do --- i before e except > after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it? It's also my pet peeve, but I long ago stopped bothering to correct people. One may also wish to consider "feisty" :) Chris
On Sun, 24 Apr 2005, Christopher Kings-Lynne wrote:
>> Permit me a digression for a pet peeve...
>>
>> "Weird" is spelled "weird". Not "wierd".
>>
>> Yes, I know the nursery rhyme as well as you do --- i before e except
>> after c, etc etc. But weird is spelled weirdly. Appropriate, isn't it?
>
> It's also my pet peeve, but I long ago stopped bothering to correct people.
I think it's quite useful to correct, because many of us use english only
when reading mailing lists and documentation :)
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
Oleg Bartunov wrote: > I think it's quite useful to correct, because many of us use english only > when reading mailing lists and documentation :) > I think that it's important to refrain from corrections on a public forum as long as the essence of the message is clear. Some people might get offended, others might just stop writing because they get intimidated by the seemingly high demands on correct spelling or sentence structure. I appreciate getting corrected by people I know in a limited forum. I would not expect it when I do a mistakes here. Can't say it ever has happend although there's often good grounds for it so I have nothing to complain about. :-) Regards, Thomas Hallgren
> I appreciate getting corrected by people I know in a limited > forum. I would not expect it when I do a mistakes here. Can't > say it ever has happend although there's often good grounds > for it so I have nothing to complain about. :-) I think you meant to say 'I Can't say it ever has happened...' :) ... John
On Sun, 24 Apr 2005, Thomas Hallgren wrote:
> Oleg Bartunov wrote:
>> I think it's quite useful to correct, because many of us use english only
>> when reading mailing lists and documentation :)
>>
> I think that it's important to refrain from corrections on a public forum as
> long as the essence of the message is clear. Some people might get offended,
> others might just stop writing because they get intimidated by the seemingly
> high demands on correct spelling or sentence structure.
>
> I appreciate getting corrected by people I know in a limited forum. I would
> not expect it when I do a mistakes here. Can't say it ever has happend
> although there's often good grounds for it so I have nothing to complain
> about. :-)
It's difficult to opposed you but "limited forum" is not we could afford.
When I see a lot of mistakes, crude words in russian forums I always feel
myself uncomfortable. Unfortunately, I can't imagine how it looks for
english speaking people. Of course, there are simple misprints which could
be painlessly ommited, but I see nothing offending against correcting
gross mistakes. Usually, people just too busy to notice them.
>
> Regards,
> Thomas Hallgren
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>
Regards, Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
Oleg Bartunov wrote: > On Sun, 24 Apr 2005, Thomas Hallgren wrote: > >> Oleg Bartunov wrote: >> >>> I think it's quite useful to correct, because many of us use >>> english only >>> when reading mailing lists and documentation :) >>> >> I think that it's important to refrain from corrections on a public >> forum as long as the essence of the message is clear. Some people >> might get offended, others might just stop writing because they get >> intimidated by the seemingly high demands on correct spelling or >> sentence structure. >> >> I appreciate getting corrected by people I know in a limited forum. I >> would not expect it when I do a mistakes here. Can't say it ever has >> happend although there's often good grounds for it so I have nothing >> to complain about. :-) > > > It's difficult to opposed you but "limited forum" is not we could afford. > When I see a lot of mistakes, crude words in russian forums I always feel > myself uncomfortable. Unfortunately, I can't imagine how it looks for > english speaking people. Of course, there are simple misprints which > could > be painlessly ommited, but I see nothing offending against correcting > gross mistakes. Usually, people just too busy to notice them. This Sentience has tree errors. Regards, Thomas Hallgren
Guys, > >> "Weird" is spelled "weird". Not "wierd". OK, spelling errors taken into account. Now could we perhaps address the **postgresql** errors? I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs. It's obviously a serious stability issue, whatever is causing it. -- Josh Berkus Aglio Database Solutions San Francisco
Josh Berkus <josh@agliodbs.com> writes:
> I'm seeing this kind of "performance plunge" on 8.1cvs in one of every 3 runs.
> It's obviously a serious stability issue, whatever is causing it.
I concur with the upthread suggestion that it may come from not doing
checkpoints in a realistic fashion, thereby allowing too much queued
I/O work to build up. What happens if you set the checkpoint interval
to 5 or 10 minutes?
regards, tom lane
Tom, > I concur with the upthread suggestion that it may come from not doing > checkpoints in a realistic fashion, thereby allowing too much queued > I/O work to build up. What happens if you set the checkpoint interval > to 5 or 10 minutes? I'll test. Keep in mind that it takes me a couple of days to run a new test (I've got 11 in the queue right now) so it'll be a bit. -- Josh Berkus Aglio Database Solutions San Francisco