Thread: Power cut and performance problem

Power cut and performance problem

From
"Daniel Caune"
Date:
Hi,

We had a power cut lastly and it seems that our PostgreSQL database
suffers from performance since.  For example, a simple query such as
"SELECT MIN(a-primary-key-column) FROM a-table" takes quite a very long
time; actually I gave up before getting the result.

I shutdown and started up the database, and I took at the log file; I
don't see any fatal error:

LOG:  database system was interrupted at 2006-03-20 22:20:22 GMT
LOG:  checkpoint record is at 10C/14919ED4
LOG:  redo record is at 10C/1487E270; undo record is at 0/0; shutdown
FALSE
LOG:  next transaction ID: 166159120; next OID: 41575
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system was not properly shut down; automatic recovery in
progress
LOG:  redo starts at 10C/1487E270
LOG:  incomplete startup packet
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
FATAL:  the database system is starting up
LOG:  record with zero length at 10C/14CF39F0
LOG:  redo done at 10C/14CF39B4
LOG:  database system is ready
LOG:  transaction ID wrap limit is 2147484146, limited by database
"postgres"
LOG:  incomplete startup packet
LOG:  received fast shutdown request
LOG:  shutting down
LOG:  database system is shut down
LOG:  database system was shut down at 2006-03-20 22:30:09 GMT
LOG:  checkpoint record is at 10C/14CF3A34
LOG:  redo record is at 10C/14CF3A34; undo record is at 0/0; shutdown
TRUE
LOG:  next transaction ID: 166159788; next OID: 41575
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system is ready
LOG:  transaction ID wrap limit is 2147484146, limited by database
"postgres"
LOG:  incomplete startup packet
LOG:  autovacuum: processing database "postgres"
LOG:  incomplete startup packet
LOG:  received fast shutdown request
LOG:  shutting down
LOG:  database system is shut down
LOG:  database system was shut down at 2006-03-20 22:31:24 GMT
LOG:  checkpoint record is at 10C/14CF3A78
LOG:  redo record is at 10C/14CF3A78; undo record is at 0/0; shutdown
TRUE
LOG:  next transaction ID: 166159796; next OID: 41575
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system is ready
LOG:  transaction ID wrap limit is 2147484146, limited by database
"postgres"
LOG:  autovacuum: processing database "postgres"
LOG:  incomplete startup packet

Where can I check, please?  Is it more likely a hardware problem (the
machine seems ok, no error detected)?

Regards,


--
Daniel CAUNE
Ubisoft Online Technology
(514) 4090 2040 ext. 5418


Re: Power cut and performance problem

From
Jeff Frost
Date:
I see you're running autovacuum.  What's your disk subsytem look like?  By 
chance is it sitting on a RAID 5 that's running in degraded mode right now 
while it scrubs?

On Mon, 20 Mar 2006, Daniel Caune wrote:

> Hi,
>
> We had a power cut lastly and it seems that our PostgreSQL database
> suffers from performance since.  For example, a simple query such as
> "SELECT MIN(a-primary-key-column) FROM a-table" takes quite a very long
> time; actually I gave up before getting the result.

-- 
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954


Re: Power cut and performance problem

From
"Daniel Caune"
Date:
> I see you're running autovacuum.  What's your disk subsytem look like?
By
> chance is it sitting on a RAID 5 that's running in degraded mode right
now
> while it scrubs?
>

Yes, that should be the problem.  I will check that tomorrow morning
with a Linux administrator.  Thanks.


Re: Power cut and performance problem

From
Jeff Frost
Date:
On Mon, 20 Mar 2006, Daniel Caune wrote:

>> I see you're running autovacuum.  What's your disk subsytem look like?
> By
>> chance is it sitting on a RAID 5 that's running in degraded mode right
> now
>> while it scrubs?
>>
>
> Yes, that should be the problem.  I will check that tomorrow morning
> with a Linux administrator.  Thanks.

BTW, I didn't complete my first thought above, which was to ask when you last 
vacuumed the DB, but then I saw that you were running autovac, so that wasn't 
likely the problem.

BTW, if the problem is actually a raid array that is rebuilding, it should be 
(hopefullY) fixed by tomorrow morning.

-- 
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954


Re: Power cut and performance problem

From
"Daniel Caune"
Date:
> BTW, I didn't complete my first thought above, which was to ask when
you
> last
> vacuumed the DB, but then I saw that you were running autovac, so that
> wasn't
> likely the problem.
>
> BTW, if the problem is actually a raid array that is rebuilding, it
should
> be
> (hopefullY) fixed by tomorrow morning.
>

An administrator is checking the raid status this morning.  Anyway, I
did some tests and it seems that some results are weird.

For example, the execution of the following query is fast as it used to
be (gslog_event_id is the primary key on gslog_event):
 select max(gslog_event_id) from gslog_event;  (=> Time: 0.773 ms)


while the following query is really slow (several minutes):
 select min(gslog_event_id) from gslog_event; (index on the primary key
is taken)


I'm not a hardware expert at all, but I supposed that the whole
performance would be degraded when a problem occurs with RAID disks.  Am
I wrong?  Could it be something else?  Are there some tools that check
the state of a PostgreSQL database?

--
Daniel


Re: Power cut and performance problem

From
Jeff Frost
Date:
On Tue, 21 Mar 2006, Daniel Caune wrote:

> For example, the execution of the following query is fast as it used to
> be (gslog_event_id is the primary key on gslog_event):
>
>  select max(gslog_event_id) from gslog_event;  (=> Time: 0.773 ms)
>
>
> while the following query is really slow (several minutes):
>
>  select min(gslog_event_id) from gslog_event; (index on the primary key
> is taken)
>
>
> I'm not a hardware expert at all, but I supposed that the whole
> performance would be degraded when a problem occurs with RAID disks.  Am
> I wrong?  Could it be something else?  Are there some tools that check
> the state of a PostgreSQL database?

You would be correct, a hardware problem should manifest itself on both those 
queries.  What is the explain analyze output of those two queries?  It's 
possible you have a corrupt index on gslog_event.  If that's the case, a 
reindex would likely remedy the problem.  Is postgres logging any errors?

-- 
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954


Re: Power cut and performance problem

From
"Daniel Caune"
Date:
> > For example, the execution of the following query is fast as it used
to
> > be (gslog_event_id is the primary key on gslog_event):
> >
> >  select max(gslog_event_id) from gslog_event;  (=> Time: 0.773 ms)
> >
> >
> > while the following query is really slow (several minutes):
> >
> >  select min(gslog_event_id) from gslog_event; (index on the primary
key
> > is taken)
> >
> >
> > I'm not a hardware expert at all, but I supposed that the whole
> > performance would be degraded when a problem occurs with RAID disks.
Am
> > I wrong?  Could it be something else?  Are there some tools that
check
> > the state of a PostgreSQL database?
>
> You would be correct, a hardware problem should manifest itself on
both
> those
> queries.  What is the explain analyze output of those two queries?
It's
> possible you have a corrupt index on gslog_event.  If that's the case,
a
> reindex would likely remedy the problem.  Is postgres logging any
errors?
>

The UNIX administrator confirms that this is not a RAID problem.

I truncate my table. This is not the most efficient way, but it's okay
because this is a data stage table.  It seems that it fixes my
performance problem.  As you said, perhaps the problem was more related
to index corruption.  Truncating data and inserting new data recreate
the index and therefore fix the problem.

Thanks,


--
Daniel


Re: Power cut and performance problem

From
"Aftab Alam"
Date:
unsubscribe

Regards,
 


-----Original Message-----
From: pgsql-sql-owner@postgresql.org
[mailto:pgsql-sql-owner@postgresql.org]On Behalf Of Daniel Caune
Sent: Tuesday, March 21, 2006 9:44 PM
To: Jeff Frost
Cc: pgsql-sql@postgresql.org
Subject: Re: [SQL] Power cut and performance problem


> > For example, the execution of the following query is fast as it used
to
> > be (gslog_event_id is the primary key on gslog_event):
> >
> >  select max(gslog_event_id) from gslog_event;  (=> Time: 0.773 ms)
> >
> >
> > while the following query is really slow (several minutes):
> >
> >  select min(gslog_event_id) from gslog_event; (index on the primary
key
> > is taken)
> >
> >
> > I'm not a hardware expert at all, but I supposed that the whole
> > performance would be degraded when a problem occurs with RAID disks.
Am
> > I wrong?  Could it be something else?  Are there some tools that
check
> > the state of a PostgreSQL database?
> 
> You would be correct, a hardware problem should manifest itself on
both
> those
> queries.  What is the explain analyze output of those two queries?
It's
> possible you have a corrupt index on gslog_event.  If that's the case,
a
> reindex would likely remedy the problem.  Is postgres logging any
errors?
> 

The UNIX administrator confirms that this is not a RAID problem.

I truncate my table. This is not the most efficient way, but it's okay
because this is a data stage table.  It seems that it fixes my
performance problem.  As you said, perhaps the problem was more related
to index corruption.  Truncating data and inserting new data recreate
the index and therefore fix the problem.

Thanks,


--
Daniel

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to      choose an index scan if your joining column's
datatypesdo not      match 

Re: Power cut and performance problem

From
"Aftab Alam"
Date:
unsubscribe

Regards,





-----Original Message-----
From: pgsql-sql-owner@postgresql.org
[mailto:pgsql-sql-owner@postgresql.org]On Behalf Of Jeff Frost
Sent: Tuesday, March 21, 2006 9:19 PM
To: Daniel Caune
Cc: pgsql-sql@postgresql.org
Subject: Re: [SQL] Power cut and performance problem


On Tue, 21 Mar 2006, Daniel Caune wrote:

> For example, the execution of the following query is fast as it used to
> be (gslog_event_id is the primary key on gslog_event):
>
>  select max(gslog_event_id) from gslog_event;  (=> Time: 0.773 ms)
>
>
> while the following query is really slow (several minutes):
>
>  select min(gslog_event_id) from gslog_event; (index on the primary key
> is taken)
>
>
> I'm not a hardware expert at all, but I supposed that the whole
> performance would be degraded when a problem occurs with RAID disks.  Am
> I wrong?  Could it be something else?  Are there some tools that check
> the state of a PostgreSQL database?

You would be correct, a hardware problem should manifest itself on both
those
queries.  What is the explain analyze output of those two queries?  It's
possible you have a corrupt index on gslog_event.  If that's the case, a
reindex would likely remedy the problem.  Is postgres logging any errors?

--
Jeff Frost, Owner     <jeff@frostconsultingllc.com>
Frost Consulting, LLC     http://www.frostconsultingllc.com/
Phone: 650-780-7908    FAX: 650-649-1954

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate      subscribe-nomail command to
majordomo@postgresql.orgso that your      message can get through to the mailing list cleanly