Thread: Performance regression on CVS head
I tried to repeat the DBT-2 runs with the "oldestxmin refresh" patch, but to my surprise the baseline run with CVS head, without the patch, behaved very differently than it did back in March. I rerun the a shorter 1h test with CVS head from May 20th, and March 6th (which is when I ran the earlier tests), and something has clearly been changed between those dates that affects the test. Test run 248 is with CVS checkout from May 20th, and 249 is from March 6th: http://community.enterprisedb.com/oldestxmin/ Vacuum on the stock table is started right after the rampup, and the drop in performance happens at the very moment that the vacuum finishes. Anyone have an explanation for this? One theory is that after VACUUM has populated the FSM, all updates need to do one extra I/O to read in a page with free space to insert to, instead of just extending the relation. But I don't think anything has changed recently in that area. Another theory is that the VACUUM updates some stats, which changes the access plan used to a much worse one. But the tables have been analyzed before the test, and again I don't remember any changes to that recently. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas <heikki@enterprisedb.com> writes:
> I tried to repeat the DBT-2 runs with the "oldestxmin refresh" patch,
> but to my surprise the baseline run with CVS head, without the patch,
> behaved very differently than it did back in March.
> I rerun the a shorter 1h test with CVS head from May 20th, and March 6th
> (which is when I ran the earlier tests), and something has clearly been
> changed between those dates that affects the test. Test run 248 is with
> CVS checkout from May 20th, and 249 is from March 6th:
May 20th is not quite my idea of "HEAD" ;-). It might be worth checking
current code before investing any think-time on this. But having said
that, it looks a bit like a planner problem --- if I'm reading the
graphs correctly, I/O wait time goes through the roof, suggesting a
change to a much less efficient plan.
regards, tom lane
Tom Lane wrote: > Heikki Linnakangas <heikki@enterprisedb.com> writes: >> I tried to repeat the DBT-2 runs with the "oldestxmin refresh" patch, >> but to my surprise the baseline run with CVS head, without the patch, >> behaved very differently than it did back in March. > >> I rerun the a shorter 1h test with CVS head from May 20th, and March 6th >> (which is when I ran the earlier tests), and something has clearly been >> changed between those dates that affects the test. Test run 248 is with >> CVS checkout from May 20th, and 249 is from March 6th: > > May 20th is not quite my idea of "HEAD" ;-). It might be worth checking > current code before investing any think-time on this. :) Yeah, I did run it with real head at first. I suspected the n_live_tuples calculations, and that's why I ran it again with a checkout from May 20th. > But having said > that, it looks a bit like a planner problem --- if I'm reading the > graphs correctly, I/O wait time goes through the roof, suggesting a > change to a much less efficient plan. Right. I'll do a "binary search" with a checkouts from different dates runs to pin it down. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Tom Lane wrote: > Heikki Linnakangas <heikki@enterprisedb.com> writes: >> I tried to repeat the DBT-2 runs with the "oldestxmin refresh" patch, >> but to my surprise the baseline run with CVS head, without the patch, >> behaved very differently than it did back in March. > >> I rerun the a shorter 1h test with CVS head from May 20th, and March 6th >> (which is when I ran the earlier tests), and something has clearly been >> changed between those dates that affects the test. Test run 248 is with >> CVS checkout from May 20th, and 249 is from March 6th: > > May 20th is not quite my idea of "HEAD" ;-). It might be worth checking > current code before investing any think-time on this. But having said > that, it looks a bit like a planner problem --- if I'm reading the > graphs correctly, I/O wait time goes through the roof, suggesting a > change to a much less efficient plan. I tracked this down to the patch to enable plan invalidation for SPI plans: http://archives.postgresql.org/pgsql-committers/2007-03/msg00136.php Apparently the vacuum causes a plan invalidation and a worse plan is chosen. I'll dig deeper into which queries are being affected and why. Unless someone has any better ideas. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas wrote: > Tom Lane wrote: >> Heikki Linnakangas <heikki@enterprisedb.com> writes: >>> I tried to repeat the DBT-2 runs with the "oldestxmin refresh" patch, >>> but to my surprise the baseline run with CVS head, without the patch, >>> behaved very differently than it did back in March. >> >>> I rerun the a shorter 1h test with CVS head from May 20th, and March >>> 6th (which is when I ran the earlier tests), and something has >>> clearly been changed between those dates that affects the test. Test >>> run 248 is with CVS checkout from May 20th, and 249 is from March 6th: >> >> May 20th is not quite my idea of "HEAD" ;-). It might be worth checking >> current code before investing any think-time on this. But having said >> that, it looks a bit like a planner problem --- if I'm reading the >> graphs correctly, I/O wait time goes through the roof, suggesting a >> change to a much less efficient plan. > > I tracked this down to the patch to enable plan invalidation for SPI plans: > > http://archives.postgresql.org/pgsql-committers/2007-03/msg00136.php > > Apparently the vacuum causes a plan invalidation and a worse plan is > chosen. I'll dig deeper into which queries are being affected and why. > Unless someone has any better ideas. Ok, found it. The plan for stock-level transaction changed as a result of a lot of dead tuples in the district table. I turned autovacuum on for the small, frequently-updated tables, and that fixed the problem. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com