Thread: Re: About "Context-switch storm" problem
We finally upgraded our database server to last PostgreSQL 8.2 available.
Since 8.2 does not include any major change (8.3 does indeed a lot of them) we had no problem with this upgrade. And believe, our database was really complex. About 600 tables with 300 stored procedures, wide use of embedded functions from different packages (including tsearch2) and lot of triggers and views. Not problems at all.
Migrating to 8.3 or 8.4 is another history. So, my recommendation is to migrate to 8.2.17
-----Original Message-----
From: 黄永卫 <yongwei.huang@gmail.com>
To: imartinez@vectorsf.com
Subject: About "Context-switch storm" problem
Date: Fri, 28 May 2010 13:31:54 +0800
Hello，Iñigo Martinez Lasala !
I read your mail about “Context-switch storm in 8.1.15” from postgres website.
I think I meet the same problem as you now .
Recently my company’s postgres DB server sluggish suddenly with a hight Context-switching value as below:
2010-04-07 04:03:15 procs memory swap io system cpu
2010-04-07 04:03:15 r b swpd free buff cache si so bi bo in cs us sy id wa
2010-04-07 14:04:27 3 0 0 2361272 272684 3096148 0 0 3 1445 973 14230 7 8 84 0
2010-04-07 14:05:27 2 0 0 2361092 272684 3096220 0 0 3 1804 1029 31852 8 10 81 1
2010-04-07 14:06:27 1 0 0 2362236 272684 3096564 0 0 3 1865 1135 19689 9 9 81 0
2010-04-07 14:07:27 1 0 0 2348400 272720 3101836 0 0 3 1582 1182 149461 15 17 67 0
2010-04-07 14:08:27 3 0 0 2392028 272840 3107600 0 0 3 3093 1275 203196 24 23 53 1
2010-04-07 14:09:27 3 1 0 2386224 272916 3107960 0 0 3 2486 1331 193299 26 22 52 0
2010-04-07 14:10:27 34 0 0 2332320 272980 3107944 0 0 3 1692 1082 214309 24 22 54 0
2010-04-07 14:11:27 1 0 0 2407432 273028 3108092 0 0 6 2770 1540 76643 29 13 57 1
2010-04-07 14:12:27 9 0 0 2358968 273104 3108388 0 0 7 2639 1466 10603 22 6 72 1
I
May I know your solution after that ?
Thank you !
Best wishes,
Ray Huang
Dear Iñigo Martinez Lasala，
Thank you for your replay and advice.
My company is a software application vendor, and our customer is useing PostgreSQL 8.1.3.
When our application is runing , the DB sluggish suddenly and application stop.
To prove that it is not the problem of the application ,We need some evidence to certified this version have problem and show to our customer.
And Can I ask you several questions?
1、 Can you recreate the "Context-switch storm" in the your old version PostgreSQL 8.1.15 ?
2、 If you can ,could you please tell me your test case ？
Thank you so much！
Best regards,
Ray Huang
发件人: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
发送时间: 2010年5月28日 15:43
收件人: 黄永卫
抄送: pgsql-admin
主题: Re: About "Context-switch storm" problem
Hi, Yongwei
We finally upgraded our database server to last PostgreSQL 8.2 available.
Since 8.2 does not include any major change (8.3 does indeed a lot of them) we had no problem with this upgrade. And believe, our database was really complex. About 600 tables with 300 stored procedures, wide use of embedded functions from different packages (including tsearch2) and lot of triggers and views. Not problems at all.
Migrating to 8.3 or 8.4 is another history. So, my recommendation is to migrate to 8.2.17
-----Original Message-----
From: 黄永卫 <yongwei.huang@gmail.com>
To: imartinez@vectorsf.com
Subject: About "Context-switch storm" problem
Date: Fri, 28 May 2010 13:31:54 +0800
Hello，Iñigo Martinez Lasala !
I read your mail about “Context-switch storm in 8.1.15” from postgres website.
I think I meet the same problem as you now .
Recently my company’s postgres DB server sluggish suddenly with a hight Context-switching value as below:
2010-04-07 04:03:15 procs memory swap io system cpu
2010-04-07 04:03:15 r b swpd free buff cache si so bi bo in cs us sy id wa
2010-04-07 14:04:27 3 0 0 2361272 272684 3096148 0 0 3 1445 973 14230 7 8 84 0
2010-04-07 14:05:27 2 0 0 2361092 272684 3096220 0 0 3 1804 1029 31852 8 10 81 1
2010-04-07 14:06:27 1 0 0 2362236 272684 3096564 0 0 3 1865 1135 19689 9 9 81 0
2010-04-07 14:07:27 1 0 0 2348400 272720 3101836 0 0 3 1582 1182 149461 15 17 67 0
2010-04-07 14:08:27 3 0 0 2392028 272840 3107600 0 0 3 3093 1275 203196 24 23 53 1
2010-04-07 14:09:27 3 1 0 2386224 272916 3107960 0 0 3 2486 1331 193299 26 22 52 0
2010-04-07 14:10:27 34 0 0 2332320 272980 3107944 0 0 3 1692 1082 214309 24 22 54 0
2010-04-07 14:11:27 1 0 0 2407432 273028 3108092 0 0 6 2770 1540 76643 29 13 57 1
2010-04-07 14:12:27 9 0 0 2358968 273104 3108388 0 0 7 2639 1466 10603 22 6 72 1
I
May I know your solution after that ?
Thank you !
Best wishes,
Ray Huang
Today database is not online due to a migration to another technology. It's not our database (a customer one) so we have no access to it at this moment.
2- One of the major on-line shops of Spain. This only happened under heavy load condition (that's under Christmas campaign), with a lot queries per minutes, some of then heavy search on database using tsearch modules. Under low load, there was no problem.
After upgrading to 8.2 and splitting our database in two (one database only for catalog searches, another database for standard operation) problem disappeared.
Our context switches where about 5000-10000 per second or less under low load scaling up to 300.000 per second under heavy load. Most of CPU time was spent on context switches and not on operating DB.
The database was 20 gigabytes size over a Quad Intel Xeon-DP (HT enabled) at 2.6Ghz with 16GB RAM memory running on a Red Hat 3.X with kernel 2.4.
First we upgraded to Red Hat 5 in order to get a kernel 2.6, then we installed postgres 8.2 and finally we splitted database.
Typical load decreased from 80-100% on high load periods to 5% or less per server with these changes. Context switches drop to 5000 or less per second.
I'm think that splitting database wouldn't have been needed, but anyway the changes were made to deal with higher loads in future... although six months later the shop was migrated to SQL Server + ASP.NET technology (decision was taken one year before and was not due to these problems :-) ).
The problem was due to several factors: kernel 2.4 + postgresql 8.2 + Intel Xeon DP
-----Original Message-----
From: 黄永卫 <yongwei.huang@gmail.com>
To: 'Iñigo Martinez Lasala' <imartinez@vectorsf.com>
Cc: 'pgsql-admin' <pgsql-admin@postgresql.org>
Subject: 答复: About "Context-switch storm" problem
Date: Mon, 31 May 2010 10:43:55 +0800
Dear Iñigo Martinez Lasala，
Thank you for your replay and advice.
My company is a software application vendor, and our customer is useing PostgreSQL 8.1.3.
When our application is runing , the DB sluggish suddenly and application stop.
To prove that it is not the problem of the application ,We need some evidence to certified this version have problem and show to our customer.
And Can I ask you several questions?
1、 Can you recreate the "Context-switch storm" in the your old version PostgreSQL 8.1.15 ?
2、 If you can ,could you please tell me your test case ？
Thank you so much！
Best regards,
Ray Huang
发送时间: 2010年5月28日 15:43
收件人: 黄永卫
抄送: pgsql-admin
主题: Re: About "Context-switch storm" problem
Hi, Yongwei
We finally upgraded our database server to last PostgreSQL 8.2 available.
Since 8.2 does not include any major change (8.3 does indeed a lot of them) we had no problem with this upgrade. And believe, our database was really complex. About 600 tables with 300 stored procedures, wide use of embedded functions from different packages (including tsearch2) and lot of triggers and views. Not problems at all.
Migrating to 8.3 or 8.4 is another history. So, my recommendation is to migrate to 8.2.17
-----Original Message-----
From: 黄永卫<yongwei.huang@gmail.com>
To: imartinez@vectorsf.com
Subject: About "Context-switch storm" problem
Date: Fri, 28 May 2010 13:31:54 +0800
Hello，Iñigo Martinez Lasala !
I read your mail about “Context-switch storm in 8.1.15” from postgres website.
I think I meet the same problem as you now .
Recently my company’s postgres DB server sluggish suddenly with a hight Context-switching value as below:
2010-04-07 04:03:15 procs memory swap io system cpu
2010-04-07 04:03:15 r b swpd free buff cache si so bi bo in cs us sy id wa
2010-04-07 14:04:27 3 0 0 2361272 272684 3096148 0 0 3 1445 973 14230 7 8 84 0
2010-04-07 14:05:27 2 0 0 2361092 272684 3096220 0 0 3 1804 1029 31852 8 10 81 1
2010-04-07 14:06:27 1 0 0 2362236 272684 3096564 0 0 3 1865 1135 19689 9 9 81 0
2010-04-07 14:07:27 1 0 0 2348400 272720 3101836 0 0 3 1582 1182 149461 15 17 67 0
2010-04-07 14:08:27 3 0 0 2392028 272840 3107600 0 0 3 3093 1275 203196 24 23 53 1
2010-04-07 14:09:27 3 1 0 2386224 272916 3107960 0 0 3 2486 1331 193299 26 22 52 0
2010-04-07 14:10:27 34 0 0 2332320 272980 3107944 0 0 3 1692 1082 214309 24 22 54 0
2010-04-07 14:11:27 1 0 0 2407432 273028 3108092 0 0 6 2770 1540 76643 29 13 57 1
2010-04-07 14:12:27 9 0 0 2358968 273104 3108388 0 0 7 2639 1466 10603 22 6 72 1
I
May I know your solution after that ?
Thank you !
Best wishes,
Ray Huang
Dear Iñigo Martinez Lasala，
Thank you for you information !
发件人: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
发送时间: 2010年5月31日 15:52
收件人: 黄永卫
抄送: 'pgsql-admin'
主题: Re: 答复: About "Context-switch storm" problem
1- Of course, we could. Problem appeared in both 8.1.8 and 8.1.15 after an initial upgrade.
Today database is not online due to a migration to another technology. It's not our database (a customer one) so we have no access to it at this moment.
2- One of the major on-line shops of Spain. This only happened under heavy load condition (that's under Christmas campaign), with a lot queries per minutes, some of then heavy search on database using tsearch modules. Under low load, there was no problem.
After upgrading to 8.2 and splitting our database in two (one database only for catalog searches, another database for standard operation) problem disappeared.
Our context switches where about 5000-10000 per second or less under low load scaling up to 300.000 per second under heavy load. Most of CPU time was spent on context switches and not on operating DB.
The database was 20 gigabytes size over a Quad Intel Xeon-DP (HT enabled) at 2.6Ghz with 16GB RAM memory running on a Red Hat 3.X with kernel 2.4.
First we upgraded to Red Hat 5 in order to get a kernel 2.6, then we installed postgres 8.2 and finally we splitted database.
Typical load decreased from 80-100% on high load periods to 5% or less per server with these changes. Context switches drop to 5000 or less per second.
I'm think that splitting database wouldn't have been needed, but anyway the changes were made to deal with higher loads in future... although six months later the shop was migrated to SQL Server + ASP.NET technology (decision was taken one year before and was not due to these problems :-) ).
The problem was due to several factors: kernel 2.4 + postgresql 8.2 + Intel Xeon DP
-----Original Message-----
From: 黄永卫 <yongwei.huang@gmail.com>
To: 'Iñigo Martinez Lasala' <imartinez@vectorsf.com>
Cc: 'pgsql-admin' <pgsql-admin@postgresql.org>
Subject: 答复: About "Context-switch storm" problem
Date: Mon, 31 May 2010 10:43:55 +0800
Dear Iñigo Martinez Lasala，
Thank you for your replay and advice.
My company is a software application vendor, and our customer is useing PostgreSQL 8.1.3.
When our application is runing , the DB sluggish suddenly and application stop.
To prove that it is not the problem of the application ,We need some evidence to certified this version have problem and show to our customer.
And Can I ask you several questions?
1、 Can you recreate the "Context-switch storm" in the your old version PostgreSQL 8.1.15 ?
2、 If you can ,could you please tell me your test case ？
Thank you so much！
Best regards,
Ray Huang
发件人: Iñigo Martinez Lasala [mailto:imartinez@vectorsf.com]
发送时间: 2010年5月28日 15:43
收件人: 黄永卫
抄送: pgsql-admin
主题: Re: About "Context-switch storm" problem
Hi, Yongwei
We finally upgraded our database server to last PostgreSQL 8.2 available.
Since 8.2 does not include any major change (8.3 does indeed a lot of them) we had no problem with this upgrade. And believe, our database was really complex. About 600 tables with 300 stored procedures, wide use of embedded functions from different packages (including tsearch2) and lot of triggers and views. Not problems at all.
Migrating to 8.3 or 8.4 is another history. So, my recommendation is to migrate to 8.2.17
-----Original Message-----
From: 黄永卫<yongwei.huang@gmail.com>
To: imartinez@vectorsf.com
Subject: About "Context-switch storm" problem
Date: Fri, 28 May 2010 13:31:54 +0800
Hello，Iñigo Martinez Lasala !
I read your mail about “Context-switch storm in 8.1.15” from postgres website.
I think I meet the same problem as you now .
Recently my company’s postgres DB server sluggish suddenly with a hight Context-switching value as below:
2010-04-07 04:03:15 procs memory swap io system cpu
2010-04-07 04:03:15 r b swpd free buff cache si so bi bo in cs us sy id wa
2010-04-07 14:04:27 3 0 0 2361272 272684 3096148 0 0 3 1445 973 14230 7 8 84 0
2010-04-07 14:05:27 2 0 0 2361092 272684 3096220 0 0 3 1804 1029 31852 8 10 81 1
2010-04-07 14:06:27 1 0 0 2362236 272684 3096564 0 0 3 1865 1135 19689 9 9 81 0
2010-04-07 14:07:27 1 0 0 2348400 272720 3101836 0 0 3 1582 1182 149461 15 17 67 0
2010-04-07 14:08:27 3 0 0 2392028 272840 3107600 0 0 3 3093 1275 203196 24 23 53 1
2010-04-07 14:09:27 3 1 0 2386224 272916 3107960 0 0 3 2486 1331 193299 26 22 52 0
2010-04-07 14:10:27 34 0 0 2332320 272980 3107944 0 0 3 1692 1082 214309 24 22 54 0
2010-04-07 14:11:27 1 0 0 2407432 273028 3108092 0 0 6 2770 1540 76643 29 13 57 1
2010-04-07 14:12:27 9 0 0 2358968 273104 3108388 0 0 7 2639 1466 10603 22 6 72 1
I
May I know your solution after that ?
Thank you !
Best wishes,
Ray Huang