Thread: pg_stat_wal_receiver and flushedUpto/writtenUpto

pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Michael Paquier
Date:
Hi,

As discussed in the thread that introduced d140f2f3 to rename
receivedUpto to flushedUpto and add writtenUpto to the WAL receiver's
shared memory information, the equivalent columns in
pg_stat_wal_receiver have not been renamed:
https://www.postgresql.org/message-id/CA+hUKGJ06d3h5JeOtAv4h52n0vG1jOPZxqMCn5FySJQUVZA32w@mail.gmail.com

When I have implemented this system view, the idea was to keep a
one-one mapping between the SQL interface and the shmem info even if
we are not compatible with past versions, hence I think that before
beta1 we had better fix that and:
- rename received_lsn to flushed_lsn.
- add one column for writtenUpto.

Attached is a patch to do that.  This needs also a catalog version
bump, and I am adding an open item.
Thanks,
--
Michael

Attachment

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Alvaro Herrera
Date:
On 2020-May-15, Michael Paquier wrote:

> As discussed in the thread that introduced d140f2f3 to rename
> receivedUpto to flushedUpto and add writtenUpto to the WAL receiver's
> shared memory information, the equivalent columns in
> pg_stat_wal_receiver have not been renamed:

> When I have implemented this system view, the idea was to keep a
> one-one mapping between the SQL interface and the shmem info even if
> we are not compatible with past versions, hence I think that before
> beta1 we had better fix that and:
> - rename received_lsn to flushed_lsn.
> - add one column for writtenUpto.

Why do you put the column at the end?  I would put written_lsn before
flushed_lsn.

Since this requires a catversion bump, I think it'd be best to do it
before beta1 next week.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Michael Paquier
Date:
On Fri, May 15, 2020 at 01:43:11PM -0400, Alvaro Herrera wrote:
> Why do you put the column at the end?  I would put written_lsn before
> flushed_lsn.

Fine by me.  I was thinking yesterday about putting the written
position after the flushed one, and finished with something that maps
with the structure.

> Since this requires a catversion bump, I think it'd be best to do it
> before beta1 next week.

Yes.  What do you think of the attached?
--
Michael

Attachment

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Alvaro Herrera
Date:
On 2020-May-16, Michael Paquier wrote:

> On Fri, May 15, 2020 at 01:43:11PM -0400, Alvaro Herrera wrote:
> > Why do you put the column at the end?  I would put written_lsn before
> > flushed_lsn.
> 
> Fine by me.  I was thinking yesterday about putting the written
> position after the flushed one, and finished with something that maps
> with the structure.

IIRC the only reason to put the written LSN where it is is so that it's
below the mutex, to indicate it's not protected by it.  Conceptually,
the written LSN is "before" the flushed LSN, which is why I propose to
put it ahead of it.

> > Since this requires a catversion bump, I think it'd be best to do it
> > before beta1 next week.
> 
> Yes.  What do you think of the attached?

Yeah, that seems good (I didn't verify the boilerplate in
pg_stat_get_wal_receiver or pg_proc.dat).  I propose

+       Last write-ahead log location already received and written to
+       disk, but not flushed.  This should not be used for data
+       integrity checks.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Michael Paquier
Date:
On Fri, May 15, 2020 at 07:34:46PM -0400, Alvaro Herrera wrote:
> IIRC the only reason to put the written LSN where it is is so that it's
> below the mutex, to indicate it's not protected by it.  Conceptually,
> the written LSN is "before" the flushed LSN, which is why I propose to
> put it ahead of it.

Sure.  My point was mainly that it is easier to compare if we are
missing any fields in the view and the order is respected.  But it
makes also sense to do things your way, so let's do that.

> Yeah, that seems good (I didn't verify the boilerplate in
> pg_stat_get_wal_receiver or pg_proc.dat).  I propose
>
> +       Last write-ahead log location already received and written to
> +       disk, but not flushed.  This should not be used for data
> +       integrity checks.

Thanks.  If there are no objections, I'll revisit that tomorrow and
apply it with those changes, just in time for beta1.
--
Michael

Attachment

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Michael Paquier
Date:
On Sat, May 16, 2020 at 10:15:47AM +0900, Michael Paquier wrote:
> Thanks.  If there are no objections, I'll revisit that tomorrow and
> apply it with those changes, just in time for beta1.

Okay, done this part then.
--
Michael

Attachment

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Fujii Masao
Date:

On 2020/05/17 10:08, Michael Paquier wrote:
> On Sat, May 16, 2020 at 10:15:47AM +0900, Michael Paquier wrote:
>> Thanks.  If there are no objections, I'll revisit that tomorrow and
>> apply it with those changes, just in time for beta1.
> 
> Okay, done this part then.

I found that "received_lsn" is still used in high-availability.sgml.
We should apply the following change in high-availability?

-     view's <literal>received_lsn</literal> indicates that WAL is being
+     view's <literal>flushed_lsn</literal> indicates that WAL is being

BTW, we have pg_last_wal_receive_lsn() that returns the same lsn as
pg_stat_wal_receiver.flushed_lsn. Previously both used the term "receive"
in their names, but currently not. IMO it's better to use the same term in
those names for the consistency, but it's not good idea to rename
pg_last_wal_receive_lsn() to something like pg_last_wal_receive_lsn().
I have no better idea for now. So I'm ok with the current names.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION



Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Michael Paquier
Date:
On Tue, May 19, 2020 at 11:38:52PM +0900, Fujii Masao wrote:
> I found that "received_lsn" is still used in high-availability.sgml.
> We should apply the following change in high-availability?
>
> -     view's <literal>received_lsn</literal> indicates that WAL is being
> +     view's <literal>flushed_lsn</literal> indicates that WAL is being

Oops, thanks.  Will fix.

> BTW, we have pg_last_wal_receive_lsn() that returns the same lsn as
> pg_stat_wal_receiver.flushed_lsn. Previously both used the term "receive"
> in their names, but currently not. IMO it's better to use the same term in
> those names for the consistency, but it's not good idea to rename
> pg_last_wal_receive_lsn() to something like pg_last_wal_receive_lsn().
> I have no better idea for now. So I'm ok with the current names.

I think you mean renaming pg_last_wal_receive_lsn() to something like
pg_last_wal_flushed_lsn(), no?  This name may become confusing because
we lose the "receive" idea in the function, that we have with the
"receiver" part of pg_stat_wal_receiver.  Maybe something like that,
though that's long:
- pg_last_wal_receive_flushed_lsn()
- pg_last_wal_receive_written_lsn()

Anyway, a rename of this function does not strike me as strongly
necessary, as that's less tied with the shared memory structure, and
we document that pg_last_wal_receive_lsn() tracks the current LSN
received and flushed.  I am actually wondering if in the future it may
not be better to remove this function, but it has no maintenance
cost either so I would just let it as-is.
--
Michael

Attachment

Re: pg_stat_wal_receiver and flushedUpto/writtenUpto

From
Fujii Masao
Date:

On 2020/05/20 8:31, Michael Paquier wrote:
> On Tue, May 19, 2020 at 11:38:52PM +0900, Fujii Masao wrote:
>> I found that "received_lsn" is still used in high-availability.sgml.
>> We should apply the following change in high-availability?
>>
>> -     view's <literal>received_lsn</literal> indicates that WAL is being
>> +     view's <literal>flushed_lsn</literal> indicates that WAL is being
> 
> Oops, thanks.  Will fix.

Thanks for the fix!

> 
>> BTW, we have pg_last_wal_receive_lsn() that returns the same lsn as
>> pg_stat_wal_receiver.flushed_lsn. Previously both used the term "receive"
>> in their names, but currently not. IMO it's better to use the same term in
>> those names for the consistency, but it's not good idea to rename
>> pg_last_wal_receive_lsn() to something like pg_last_wal_receive_lsn().
>> I have no better idea for now. So I'm ok with the current names.
> 
> I think you mean renaming pg_last_wal_receive_lsn() to something like
> pg_last_wal_flushed_lsn(), no?

No, that's not good idea, as I told upthread.

> This name may become confusing because
> we lose the "receive" idea in the function, that we have with the
> "receiver" part of pg_stat_wal_receiver.  Maybe something like that,
> though that's long:
> - pg_last_wal_receive_flushed_lsn()
> - pg_last_wal_receive_written_lsn()

Yes, that's long.

> Anyway, a rename of this function does not strike me as strongly
> necessary, as that's less tied with the shared memory structure, and
> we document that pg_last_wal_receive_lsn() tracks the current LSN
> received and flushed.  I am actually wondering if in the future it may
> not be better to remove this function, but it has no maintenance
> cost either so I would just let it as-is.

Yeah, agreed.

Regards,

-- 
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION