Re: Disable WAL logging to speed up data loading - Mailing list pgsql-hackers

From Kyotaro Horiguchi
Subject Re: Disable WAL logging to speed up data loading
Date
Msg-id 20201127.153843.1190863187027656694.horikyota.ntt@gmail.com
Whole thread Raw
In response to Re: Disable WAL logging to speed up data loading  (Masahiko Sawada <sawada.mshk@gmail.com>)
List pgsql-hackers
At Fri, 27 Nov 2020 15:07:34 +0900, Masahiko Sawada <sawada.mshk@gmail.com> wrote in 
> On Tue, Nov 24, 2020 at 11:19 AM osumi.takamichi@fujitsu.com
> <osumi.takamichi@fujitsu.com> wrote:
> >
> > Hi
> >
> > On Monday, Nov 23, 2020 12:17 PM Tsunakawa, Takayuki <tsunakawa.takay@fujitsu.com> wrote:
> > > PREPARE TRANSACTION is the same as COMMIT in that it persists
> > > transaction updates.  A crash during wal_level = none loses both of them.
> > > So, I don't think PREPARE TRANSACTION needs special treatment.
> > Yeah, I got it. That makes sense.
> > Attached is the one I removed the special treatment.
> >
> >
> > > Does PREPARE TRANSACTION complete successfully?  I remember
> > > Fujii-san said it PANICed if --enable-asserts is used in configure.
> > Yes. That assertion failure Fujii-San reported in the past
> > was solved by having rmid != RM_XACT_ID in XLogInsert()
> > to add WAL generation for transaction completes.
> > That I missed the condition was the cause but fine now.
> >
> > Plus, PREPARE TRANSACTION and COMMIT PREPARED
> > works successfully at present when no happening occurs.
> >
> 
> Thank you for updating the patch.
> 
> -               (errmsg("WAL was generated with wal_level=minimal,
> data may be missing"),
> +               (errmsg("WAL was generated with wal_level<=minimal,
> data may be missing"),
>                  errhint("This happens if you temporarily set
> wal_level=minimal without taking a new base backup.")));
> 
> 'wal_level=minimal' in errhint also needs to be changed to 'wal_level<=minimal'?
> 
> While testing the patch on some workload, I realized that
> XLOG_FPI_FOR_HINT record could still be emitted even when wal_level =
> none. IIUC that WAL record is not necessary during wal_level = none
> since the server cannot be the primary server and the server crash
> ends up requiring to restore the whole database.

It seems to be on purpose.

@@ -449,6 +449,14 @@ XLogInsert(RmgrId rmid, uint8 info)
         return EndPos;
     }
 
+    /* Issues WAL related to XLOG resources and transactions only */
+    if (wal_level == WAL_LEVEL_NONE &&
+        rmid != RM_XLOG_ID && rmid != RM_XACT_ID)
+    {
+        XLogResetInsertion();
+        return GetXLogInsertRecPtr();

What is the reason for those kinds of records to be emitted?

I think we must emit at least the shutdown checkpoint record, which
has redo-LSN that points to the record itself.  I'm not sure what
other kinds of records are needed.

regards.

-- 
Kyotaro Horiguchi
NTT Open Source Software Center



pgsql-hackers by date:

Previous
From: "Hou, Zhijie"
Date:
Subject: RE: Parallel Inserts in CREATE TABLE AS
Next
From: Luc Vlaming
Date:
Subject: Re: Multi Inserts in CREATE TABLE AS - revived patch