Re: PG 14 release notes, first draft - Mailing list pgsql-hackers

On Tue, May 11, 2021 at 09:26:36AM +0530, Amit Kapila wrote:
> On Mon, May 10, 2021 at 7:30 PM Justin Pryzby <pryzby@telsasoft.com> wrote:
> >
> > Can these be merged:
> >  Allow logical replication to stream long transactions to standbys (Dilip Kumar, Tomas Vondra, Amit Kapila, Nikhil
Sontakke)
> >  Improve the logical replication API to allow streaming large in-progress transactions (Tomas Vondra, Dilip Kumar,
AmitKapila)
 
> >
> 
> I think we should merge the below two:
> Add support for streaming to built-in logical replication (Vondra,
> Dilip Kumar, Amit Kapila, Ajin Cherian, Nikhil Sontakke, Stas Kelvich)
> 
> Allow logical replication to stream long transactions to standbys
> (Dilip Kumar, Tomas Vondra, Amit Kapila, Nikhil Sontakke)
> 
> and say something like:
> Allow logical replication to stream long in-progress transactions to standbys
> Previously transactions that exceeded logical_decoding_work_mem were
> written to disk until the transaction completed.

OK, updated as you suggested.

> Also, the commits that lead to the above feature is:
> 2020-09-03 [464824323] Add support for streaming to built-in logical replicatio
> 2020-08-08 [7259736a6] Implement streaming mode in ReorderBuffer.

Yes, these were already listed above that item.

> * Improve the logical replication API to allow streaming large
> in-progress transactions (Tomas Vondra, Dilip Kumar, Amit Kapila)
> The output functions begin with "stream". test_decoding also supports these.
> 
> In the above, I think it is better to say Enhance instead of Improve.

OK, word changed.

> * Add system view pg_stat_replication_slots to report replication slot
> activity (Sawada Masahiko, Amit Kapila)
> 
> I think for this item, we can give Vignesh C as author credit because
> he helped a lot enhance/stabilize this feature after the initial
> commit. See
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=f5fc2f5b23d1b1dff60f8ca5dc211161df47eda4
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=cca57c1d9bf7eeba5b81115e0b82651cf3d8e4ea
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=592f00f8dec68038301467a904ac514eddabf6cd

OK, I have added him:

    Add system view pg_stat_replication_slots to report replication slot
    activity (Sawada Masahiko, Amit Kapila, Vignesh C)

> * Allow multiple xacts during table sync in logical replication (Peter
> Smith, Amit Kapila, and Takamichi Osumi)
> 
> IMPORTANT?
> 
> Yes, this is a big enough improvement in the initial table sync
> mechanism of logical replication that it is worth mentioning in
> Release Notes. One of the main user-visible effects this commit can
> have is that it will reduce the time for which we need to hold WAL on
> the server during the initial table sync phase. Then it can avoid
> large copies in case there is any error during the time when
> apply-worker and tablesync-worker synchronizes the apply location.

OK, thanks.

> * Enable logical replication to handle two phase commits (Ajin Cherian)
> This is controlled via pg_create_logical_replication_slot().
> 
> We still haven't supported two-phase commits for built-in logical
> replication but we have exposed APIs for decoding plugins to achieve
> that. Using these APIs, we are planning to support it for built-in
> logical replication but that will be probably done in PG-15. So, I
> suggest we can modify it to something like:
> 
> * Enhance logical decoding APIs to handle two-phase commits
> This is controlled via pg_create_logical_replication_slot().

OK, updated as you suggested.

> Most of the work for this feature is done via commits:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=a271a1b50e9bec07e2ef3a05e38e7285113e4ce6
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0aa8a01d04c8fe200b7a106878eebc3d0af9105c
> 
> So, I think the author credit should be given to Ajin Cherian, Amit
> Kapila, Nikhil Sontakke, and Stas Kelvich

OK, the new item text is:

    <listitem>
    <!--
    Author: Amit Kapila <akapila@postgresql.org>
    2020-12-30 [0aa8a01d0] Extend the output plugin API to allow decoding of prepar
    Author: Amit Kapila <akapila@postgresql.org>
    2021-01-04 [a271a1b50] Allow decoding at prepare time in ReorderBuffer.
    Author: Amit Kapila <akapila@postgresql.org>
    2021-03-03 [19890a064] Add option to enable two_phase commits via pg_create_log
    -->
    
    <para>
    Enhance logical decoding APIs to handle two-phase commits (Ajin
    Cherian, Amit Kapila, Nikhil Sontakke, Stas Kelvich)
    </para>
    
    <para>
    This is controlled via pg_create_logical_replication_slot().
    </para>
    </listitem>

> * I think we should mention about commit:
> https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=d7eb52d7181d83cf2363570f7a205b8eb1008dbc
> 
> This allows Logical decoding to work efficiently in presence of a
> large amount of DDLs. This was a complaint from many users for long
> time. Apart from the discussion thread, see some of the earlier
> complaints:
> https://www.postgresql.org/message-id/flat/20f3de7675f83176253f607b5e199b228406c21c.camel@cybertec.at
> https://www.postgresql.org/message-id/flat/CAHoiPjzea6N0zuCi=+f9v_j94nfsy6y8SU7-=bp4=7qw6_i=Rg@mail.gmail.com
> 
> and most recently at:
>
https://www.postgresql.org/message-id/DB8PR10MB36126BC0DC3570637FDFED12EB4F9%40DB8PR10MB3612.EURPRD10.PROD.OUTLOOK.COM

OK, I added this:

    <listitem>
    <!--
    Author: Amit Kapila <akapila@postgresql.org>
    2020-10-15 [d7eb52d71] Execute invalidation messages for each XLOG_XACT_INVALID
    -->
    
    <para>
    Allow logical decoding to more efficently process cache invalidation messages
    </para>
    
    <para>
    This allows Logical decoding to work efficiently in presence of a large amount of DDL.
    </para>
    </listitem>

> * Speed truncation of small tables on large shared buffer servers (Kirk Jamison)
> 
> This optimization applies during recovery. Shouldn't we mention that
> in some way?

Yes, new text:

    Speed truncation of small tables during recovery on clusters with a
    large number of shared buffers (Kirk Jamison)

-- 
  Bruce Momjian  <bruce@momjian.us>        https://momjian.us
  EDB                                      https://enterprisedb.com

  If only the physical world exists, free will is an illusion.