Re: .ready and .done files considered harmful - Mailing list pgsql-hackers

From Dipesh Pandit
Subject Re: .ready and .done files considered harmful
Date
Msg-id CAN1g5_Gqz4t2QvHx7UOxfz9sJ_9fXE+KXFiW_u9KxoMHOkypOw@mail.gmail.com
Whole thread Raw
In response to Re: .ready and .done files considered harmful  (Jeevan Ladhe <jeevan.ladhe@enterprisedb.com>)
Responses Re: .ready and .done files considered harmful
List pgsql-hackers
Hi,

> I agree, I missed this part. The .history file should be given higher preference.
> I will take care of it in the next patch.

Archiver does not have access to shared memory and the current timeline ID
is not available at archiver. In order to keep track of timeline switch we have
to push a notification from backend to archiver.  Backend can send a signal
to notify archiver about the timeline change. Archiver can register this
notification and perform a full directory scan to make sure that archiving
history files take precedence over archiving WAL files.

> If a history file is found we are not updating curFileTLI and
> nextLogSegNo, so it will attempt the previously found segment.  This
> is fine because it will not find that segment and it will rescan the
> directory.  But I think we can do better, instead of searching the
> same old segment in the previous timeline we can search that old
> segment in the new TL so that if the TL switch happened within the
> segment then we will find the segment and we will avoid the directory
> search.

This could have been done with the approach mentioned in patch v1 but now
considering archiving history file takes precedence over WAL files we cannot
update the "curFileTLI" whenever a history file is found.

> So everytime archiver will start with searching segno=0 in timeline=0.
> Instead of doing this can't we first scan the directory and once we
> get the first segment to archive then only we can start predicting the
> next wal segment?

Done.

> This comment is a bit confusing with the name of the variable nextLogSegNo.
> I think the name of the variable is appropriate here, but maybe we can reword
> the comment something like:

Done.

I have incorporated these changes and updated a new patch. PFA, patch v2.

Thanks,
Dipesh
Attachment

pgsql-hackers by date:

Previous
From: "houzj.fnst@fujitsu.com"
Date:
Subject: RE: Skipping logical replication transactions on subscriber side
Next
From: John Naylor
Date:
Subject: Re: O_DIRECT on macOS