remote_apply commit hangs when wal_receiver_status_interval = 0 - Mailing list pgsql-hackers

Hi hackers,

On master, a commit with synchronous_commit = remote_apply does not
return when the standby has wal_receiver_status_interval = 0. The
standby applies the commit, but it never sends the apply reply to the
primary, so the backend keeps waiting in SyncRep. With default
timeouts it waits about 30 seconds, until the walsender keepalive
forces a reply. With wal_sender_timeout = 0 and wal_receiver_timeout
= 0 it waits forever.

The docs for wal_receiver_status_interval say that updates are sent
while ignoring this parameter "when synchronous_commit is set to
remote_apply", so I think this is a regression.

Steps to reproduce:

     initdb -D pg20_primary -U postgres -A trust
    cat >> pg20_primary/postgresql.conf <<EOF
    port = 5521
    wal_sender_timeout = 0
    EOF
    pg_ctl -D pg20_primary -l primary.log -w st
    psql -p 5521 -U postgres -c "CREATE TABLE t (id int, note text);"

    pg_basebackup -h 127.0.0.1 -p 5521 -U postgres -D pg20_standby -R -d "application_name=standby_5522"
    cat >> pg20_standby/postgresql.conf <<EOF
    port = 5522
    wal_receiver_status_interval = 0
    wal_receiver_timeout = 0
    hot_standby_feedback = off
    EOF
    pg_ctl -D pg20_standby -l standby.log -w start

    psql -p 5521 -U postgres \
        -c "ALTER SYSTEM SET synchronous_standby_names = 'standby_5522';"
        -c "SELECT pg_reload_conf();"

The standby is streaming and shows as sync:

     application_name |   state   | sync_state
    ------------------+-----------+------------
     standby_5522     | streaming | sync

Now run a remote_apply commit on the primary:

     postgres=# SET synchronous_commit = remote_apply;
    postgres=# \timing on
    postgres=# INSERT INTO t VALUES (1, 'remote_apply');
    ^CCancel request sent
    WARNING:  canceling wait for synchronous replication due to user request
    DETAIL:  The transaction has already committed locally, but might
    not have been replicated to the standby.
    INSERT 0 1
    Time: 184888.783 ms (03:04.889)

While the INSERT was stuck, the backend was waiting in SyncRep:

       pid   | wait_event_type | wait_event |     waiting
    ---------+-----------------+------------+--
     2947177 | IPC             | SyncRep    | 00:00:51.399588

But the row was already visible on the standby:

     id |     note
    ----+--------------
      1 | remote_apply

The primary's replay_lsn in pg_stat_replication stays at the value
from connection time. So the standby has applied the commit, but the
primary never learns about it.

Root cause:

Commit 400a790a48e changed the apply notification call in the
walreceiver main loop from

    XLogWalRcvSendReply(true, false);

to

    XLogWalRcvSendReply(false, false, true);  

But XLogWalRcvSendReply() still has this early return at the top:

    if (!force && wal_receiver_status_interval <=0)
        return;

So with the interval set to 0, the apply reply is dropped before the
new checkApply logic is reached. The same happens for the reply