Re: pgstat wait timeout (RE: contrib/cache_scan) - Mailing list pgsql-hackers

From Tomas Vondra
Subject Re: pgstat wait timeout (RE: contrib/cache_scan)
Date
Msg-id 49fa6cf98701d2529cbe5ec18e342bde.squirrel@sq.gransy.com
Whole thread Raw
In response to pgstat wait timeout (RE: contrib/cache_scan)  (Kouhei Kaigai <kaigai@ak.jp.nec.com>)
List pgsql-hackers
On 12 Březen 2014, 14:54, Kouhei Kaigai wrote:
> It is another topic from the main thread,
>
> I noticed the following message under the test cases that
> takes heavy INSERT workload; provided by Haribabu.
>
> [kaigai@iwashi ~]$ createdb mytest
> [kaigai@iwashi ~]$ psql -af ~/cache_scan.sql mytest
> \timing
> Timing is on.
> --cache scan select 5 million
> create table test(f1 int, f2 char(70), f3 float, f4 char(100));
> CREATE TABLE
> Time: 22.373 ms
> truncate table test;
> TRUNCATE TABLE
> Time: 17.705 ms
> insert into test values (generate_series(1,5000000), 'fujitsu', 1.1,
> 'Australia software tech pvt ltd');
> WARNING:  pgstat wait timeout
> WARNING:  pgstat wait timeout
> WARNING:  pgstat wait timeout
> WARNING:  pgstat wait timeout
>    :
>
> Once I got above messages, write performance is dramatically
> degraded, even though I didn't take detailed investigation.
>
> I could reproduce it on the latest master branch without my
> enhancement, so I guess it is not a problem something special
> to me.
> One other strangeness is, right now, this problem is only
> happen on my virtual machine environment - VMware ESXi 5.5.0.
> I couldn't reproduce the problem on my physical environment
> (Fedora20, core i5-4570S).
> Any ideas?

I've seen this happening in cases when it was impossible to write
the stat file for some reason. IIRC there were two basic causes I've seen
in the past:

(1) writing the stat copy failed - for example when the temporary stat
directory was placed in tmpfs, but it was too small

(2) writing the stat copy took too long - e.g. with tmpfs and memory
pressure, forcing the system to swap to free space for the stat copy

(3) IIRC the inquiry (backend -> postmaster) to write the file is sent
using UDP, which may be dropped in some cases (e.g. when the system is
overloaded), so the postmaster does not even know it should write the file

I'm not familiar with VMware ESXi virtualization, but I suppose it might
be relevant to all three causes.

regards
Tomas




pgsql-hackers by date:

Previous
From: Stephen Frost
Date:
Subject: Re: db_user_namespace a "temporary measure"
Next
From: Tom Lane
Date:
Subject: Re: COPY table FROM STDIN doesn't show count tag