Thread: pgbench -i spends all its time doing CHECKPOINT
Could we remove lines 552-560 of pgbench.c? The behavior that guarded
against is long gone, and forcing a checkpoint every few thousand tuples
seems to be putting a huge crimp in the speed of pgbench -i ...
regards, tom lane
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> Could we remove lines 552-560 of pgbench.c? The behavior that guarded
>> against is long gone, and forcing a checkpoint every few thousand tuples
>> seems to be putting a huge crimp in the speed of pgbench -i ...
> I don't see any need for it.
Au contraire. I'm nearly done with pgbench -i -s 500 after about an
hour. The unmodified code ran for several hours and was less than
half done.
regards, tom lane
Tom Lane wrote: > Could we remove lines 552-560 of pgbench.c? The behavior that guarded > against is long gone, and forcing a checkpoint every few thousand tuples > seems to be putting a huge crimp in the speed of pgbench -i ... I don't see any need for it. It isn't even needed for 7.1.3. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > Tom Lane wrote: > >> Could we remove lines 552-560 of pgbench.c? The behavior that guarded > >> against is long gone, and forcing a checkpoint every few thousand tuples > >> seems to be putting a huge crimp in the speed of pgbench -i ... > > > I don't see any need for it. > > Au contraire. I'm nearly done with pgbench -i -s 500 after about an > hour. The unmodified code ran for several hours and was less than > half done. I meant I don't see any need for that checkpoint code. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I meant I don't see any need for that checkpoint code.
Oh, my mistake; I thought you meant you didn't want to make a change
now.
regards, tom lane
> Could we remove lines 552-560 of pgbench.c? The behavior that guarded > against is long gone, and forcing a checkpoint every few thousand tuples > seems to be putting a huge crimp in the speed of pgbench -i ... Yup. Maybe we could ifdef'ed out until we implement true UNDO... -- Tatsuo Ishii
Tatsuo Ishii wrote: > > Could we remove lines 552-560 of pgbench.c? The behavior that guarded > > against is long gone, and forcing a checkpoint every few thousand tuples > > seems to be putting a huge crimp in the speed of pgbench -i ... > > Yup. Maybe we could ifdef'ed out until we implement true UNDO... I think we should just remove it. The idea that we are going to do UNDO which allows unlimited log file growth for long transactions seems like a loser to me. Actually, that brings up a question I had. In 7.1.0, we didn't recycle WAL segements that were used by open transactions during CHECKPOINT, while in 7.1.3 and later, we do recycle them after CHECKPOINT. My question is if we do a big transaction that needs 10 log segments, do we force an early CHECKPOINT to clear out the WAL segments or do we just wait for the proper interval? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian <pgman@candle.pha.pa.us> writes: > question is if we do a big transaction that needs 10 log segments, do we > force an early CHECKPOINT to clear out the WAL segments or do we just > wait for the proper interval? A checkpoint is forced after every CHECKPOINT_SEGMENTS log segments, regardless of longevity of transactions. See http://developer.postgresql.org/docs/postgres/wal-configuration.html Since segments before the checkpoint-before-last are deleted or recycled after each checkpoint, the maximum number of back segments would normally be 2 * CHECKPOINT_SEGMENTS. We also pre-create WAL_FILES future log segments. Counting the current segment gives a total of WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments. AFAICS, the only way to force the current code into creating more than WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments is to be generating WAL entries at such a high rate that more than WAL_FILES log segments are filled before a triggered checkpoint can be completed. regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > question is if we do a big transaction that needs 10 log segments, do we > > force an early CHECKPOINT to clear out the WAL segments or do we just > > wait for the proper interval? > > A checkpoint is forced after every CHECKPOINT_SEGMENTS log segments, > regardless of longevity of transactions. See > http://developer.postgresql.org/docs/postgres/wal-configuration.html > > Since segments before the checkpoint-before-last are deleted or recycled > after each checkpoint, the maximum number of back segments would > normally be 2 * CHECKPOINT_SEGMENTS. We also pre-create WAL_FILES > future log segments. Counting the current segment gives a total of > WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments. > > AFAICS, the only way to force the current code into creating more than > WAL_FILES + 2 * CHECKPOINT_SEGMENTS + 1 log segments is to be generating > WAL entries at such a high rate that more than WAL_FILES log segments > are filled before a triggered checkpoint can be completed. > Very interesting. Thanks. Is there a reason someone would manually run the CHECKPOINT command? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026