Re: checkpointer continuous flushing - Mailing list pgsql-hackers

From Amit Kapila
Subject Re: checkpointer continuous flushing
Date
Msg-id CAA4eK1KK6ab4epZbs=_YLHTCNDmtCdPPzQw7cYcq=Mrd6SwOGw@mail.gmail.com
Whole thread Raw
In response to Re: checkpointer continuous flushing  (Fabien COELHO <coelho@cri.ensmp.fr>)
Responses Re: checkpointer continuous flushing  (Fabien COELHO <coelho@cri.ensmp.fr>)
List pgsql-hackers
On Sun, Aug 23, 2015 at 12:33 PM, Fabien COELHO <coelho@cri.ensmp.fr> wrote:

Hello Amit,

I have tried your scripts and found some problem while using avg.py
script.
grep 'progress:' test_medium4_FW_off.out | cut -d' ' -f4 | ./avg.py
--limit=10 --length=300
: No such file or directory

I didn't get chance to poke into avg.py script (the command without
avg.py works fine). Python version on the m/c, I planned to test is
Python 2.7.5.

Strange... What does "/usr/bin/env python" say?

Python 2.7.5 (default, Apr  9 2015, 11:07:29) 
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 
 
Can the script be started on its own at all?

I have tried like below which results in same error, also I tried few
other variations but could not succeed.
./avg.py 
: No such file or directory
 

Here, I think above calculation can go for toss if backend or bgwriter
starts writing buffers when checkpoint is in progress.  The tablespace
written parameter won't be able to consider the one's written by backends
or bgwriter.

Sure... This is *already* the case with the current checkpointer, the schedule is performed with respect to the initial number of buffers it think it will have to write, and if someone else writes these buffers then the schedule is skewed a little bit, or more... I have not changed this logic, but I extended it to handle several tablespaces.


I don't know how good or bad it is to build  further on somewhat skewed
logic, but the point is that unless it is required why to use it.
 
I do not think that Heikki version worked wrt to balancing writes over tablespaces,

I also think that it doesn't balances over tablespaces, but the question
is why do we need to balance over tablespaces, can we reliably
predict in someway which indicates that performing balancing over
tablespace can help the workload.  I think here we are doing more
engineering than required for this patch.  
 
and I'm not sure it worked at all.

Okay, his version might have some bugs, but then those could be
fixed as well.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com

pgsql-hackers by date:

Previous
From: Michael Paquier
Date:
Subject: Re: Archiving done right
Next
From: Pavel Stehule
Date:
Subject: psql - better support pipe line