Thread: [HACKERS] Logical replication - TRAP: FailedAssertion in pgstat.c
Testing logical replication, with the following patches on top of
yesterday's master:
0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch +
0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+
0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch +
0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch +
0005-Skip-unnecessary-snapshot-builds.patch
Is applying that patch set is still correct?
It builds fine, but when I run the old pbench-over-logical-replication
test I get:
TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File:
"pgstat.c", Line: 828)
reliably (often within a minute).
The test itself does not fail, at least not that I saw (but I only ran a
few).
thanks,
Erik Rijkers
On 2017-04-15 04:47, Erik Rijkers wrote: > > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + > 0005-Skip-unnecessary-snapshot-builds.patch I am now using these newer patches: https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com > It builds fine, but when I run the old pbench-over-logical-replication > test I get: > > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: > "pgstat.c", Line: 828) To get that error: -------------- #!/bin/sh port1=6972 port2=6973 scale=25 clients=16 duration=60 echo "drop table if exists pgbench_accounts; drop table if exists pgbench_branches; drop table if existspgbench_tellers; drop table if exists pgbench_history;" | psql -qXp $port1 \ && echo "drop table if exists pgbench_accounts; drop table if exists pgbench_branches; drop table if existspgbench_tellers; drop table if exists pgbench_history;" | psql -qXp $port2 \ && pgbench -p $port1 -qis ${scale//_/} && echo " alter table pgbench_history add column hid serial primary key; " | psql -q1Xp $port1 \ && pg_dump -F c -p $port1 \ --exclude-table-data=pgbench_history \ --exclude-table-data=pgbench_accounts\ --exclude-table-data=pgbench_branches \ --exclude-table-data=pgbench_tellers \ -t pgbench_history \ -t pgbench_accounts \ -t pgbench_branches\ -t pgbench_tellers \ | pg_restore -1 -p $port2 -d testdb appname=pgbench_derail echo "create publication pub1 for all tables;" | psql -p $port1 -aqtAX echo "create subscription sub1 connection 'port=${port1} application_name=${appname}' publication pub1 with (disabled); alter subscription sub1 enable; " | psql -p $port2 -aqtAX echo "-- pgbench -p $port1 -c $clients -T $duration -n -- scale $scale " pgbench -p $port1 -c $clients -T $duration -n -------------- Erik Rijkers
On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote: > On 2017-04-15 04:47, Erik Rijkers wrote: > > > > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + > > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ > > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + > > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + > > 0005-Skip-unnecessary-snapshot-builds.patch > > I am now using these newer patches: > https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com > > > It builds fine, but when I run the old pbench-over-logical-replication > > test I get: > > > > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: > > "pgstat.c", Line: 828) > > > To get that error: I presume this is the fault of http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746 if you git revert that individual commit, do things work again? - Andres
On 16/04/17 20:41, Andres Freund wrote: > On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote: >> On 2017-04-15 04:47, Erik Rijkers wrote: >>> >>> 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + >>> 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ >>> 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + >>> 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + >>> 0005-Skip-unnecessary-snapshot-builds.patch >> >> I am now using these newer patches: >> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com >> >>> It builds fine, but when I run the old pbench-over-logical-replication >>> test I get: >>> >>> TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: >>> "pgstat.c", Line: 828) >> >> >> To get that error: > > I presume this is the fault of > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746 > if you git revert that individual commit, do things work again? > > - Andres > Yeah it is, it needs to be fenced to happen only after commit, which is not guaranteed at the point of code, we probably need to put the pgstat_report_stat() inside the if above after the CommitTransactionCommand() (that will make it report stats for changes apply did to pg_subscription_rel after next transaction though) or put it into it's own if that checks if tx was indeed committed (which is bit ugly). -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 2017-04-16 20:41, Andres Freund wrote: > On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote: >> On 2017-04-15 04:47, Erik Rijkers wrote: >> > >> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + >> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ >> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + >> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + >> > 0005-Skip-unnecessary-snapshot-builds.patch >> >> I am now using these newer patches: >> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com >> >> > It builds fine, but when I run the old pbench-over-logical-replication >> > test I get: >> > >> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: >> > "pgstat.c", Line: 828) >> >> >> To get that error: > > I presume this is the fault of > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746 > if you git revert that individual commit, do things work again? > Yes, compiled from 67c2def11d4 with the above 4 patches, it runs flawlessly again. (flawlessly= a few hours without any error)
> On 17 Apr 2017, at 10:30, Erik Rijkers <er@xs4all.nl> wrote: > > On 2017-04-16 20:41, Andres Freund wrote: >> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote: >>> On 2017-04-15 04:47, Erik Rijkers wrote: >>> > >>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + >>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ >>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + >>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + >>> > 0005-Skip-unnecessary-snapshot-builds.patch >>> I am now using these newer patches: >>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com >>> > It builds fine, but when I run the old pbench-over-logical-replication >>> > test I get: >>> > >>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: >>> > "pgstat.c", Line: 828) >>> To get that error: >> I presume this is the fault of >> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746 >> if you git revert that individual commit, do things work again? > > Yes, compiled from 67c2def11d4 with the above 4 patches, it runs flawlessly again. (flawlessly= a few hours without anyerror) > I’ve reproduced failure, this happens under tablesync worker and putting pgstat_report_stat() under the previous condition block should help. However for me it took about an hour of running this script to catch original assert. Can you check with that patch applied? Stas Kelvich Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On 2017-04-17 15:59, Stas Kelvich wrote: >> On 17 Apr 2017, at 10:30, Erik Rijkers <er@xs4all.nl> wrote: >> >> On 2017-04-16 20:41, Andres Freund wrote: >>> On 2017-04-16 10:46:21 +0200, Erik Rijkers wrote: >>>> On 2017-04-15 04:47, Erik Rijkers wrote: >>>> > >>>> > 0001-Reserve-global-xmin-for-create-slot-snasphot-export.patch + >>>> > 0002-Don-t-use-on-disk-snapshots-for-snapshot-export-in-l.patch+ >>>> > 0003-Prevent-snapshot-builder-xmin-from-going-backwards.patch + >>>> > 0004-Fix-xl_running_xacts-usage-in-snapshot-builder.patch + >>>> > 0005-Skip-unnecessary-snapshot-builds.patch >>>> I am now using these newer patches: >>>> https://www.postgresql.org/message-id/30242bc6-eca4-b7bb-670e-8d0458753a8c%402ndquadrant.com >>>> > It builds fine, but when I run the old pbench-over-logical-replication >>>> > test I get: >>>> > >>>> > TRAP: FailedAssertion("!(entry->trans == ((void *)0))", File: >>>> > "pgstat.c", Line: 828) >>>> To get that error: >>> I presume this is the fault of >>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=139eb9673cb84c76f493af7e68301ae204199746 >>> if you git revert that individual commit, do things work again? >> >> Yes, compiled from 67c2def11d4 with the above 4 patches, it runs >> flawlessly again. (flawlessly= a few hours without any error) >> > > I’ve reproduced failure, this happens under tablesync worker and > putting > pgstat_report_stat() under the previous condition block should help. > > However for me it took about an hour of running this script to catch > original assert. > > Can you check with that patch applied? Your patch on top of the 5 patches above seem to solve the matter too: no problems after running for 2 hours (previously it failed within half a minute). Erik Rijkers
On 4/16/17 16:11, Petr Jelinek wrote: > Yeah it is, it needs to be fenced to happen only after commit, which is > not guaranteed at the point of code, we probably need to put the > pgstat_report_stat() inside the if above after the > CommitTransactionCommand() (that will make it report stats for changes > apply did to pg_subscription_rel after next transaction though) I think to avoid the latter, we should add more pgstat_report_stat() calls, such as in process_syncing_tables_for_apply(). Basically every code path that calls CommitTransactionCommand() should have one, no? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> wrote: > On 4/16/17 16:11, Petr Jelinek wrote: >> Yeah it is, it needs to be fenced to happen only after commit, which is >> not guaranteed at the point of code, we probably need to put the >> pgstat_report_stat() inside the if above after the >> CommitTransactionCommand() (that will make it report stats for changes >> apply did to pg_subscription_rel after next transaction though) > > I think to avoid the latter, we should add more pgstat_report_stat() > calls, such as in process_syncing_tables_for_apply(). Basically every > code path that calls CommitTransactionCommand() should have one, no? Is there anything left to be committed here? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 02/05/17 20:43, Robert Haas wrote: > On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > <peter.eisentraut@2ndquadrant.com> wrote: >> On 4/16/17 16:11, Petr Jelinek wrote: >>> Yeah it is, it needs to be fenced to happen only after commit, which is >>> not guaranteed at the point of code, we probably need to put the >>> pgstat_report_stat() inside the if above after the >>> CommitTransactionCommand() (that will make it report stats for changes >>> apply did to pg_subscription_rel after next transaction though) >> >> I think to avoid the latter, we should add more pgstat_report_stat() >> calls, such as in process_syncing_tables_for_apply(). Basically every >> code path that calls CommitTransactionCommand() should have one, no? > > Is there anything left to be committed here? > Afaics the fix was not committed. Peter wanted more comprehensive fix which didn't happen. I think something like attached should do the job. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On 2017-05-03 08:17, Petr Jelinek wrote: > On 02/05/17 20:43, Robert Haas wrote: >> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut >>> code path that calls CommitTransactionCommand() should have one, no? >> >> Is there anything left to be committed here? >> > > Afaics the fix was not committed. Peter wanted more comprehensive fix > which didn't happen. I think something like attached should do the job. I'm running my pgbench-over-logical-replication test in chunk of 15 minutes, wth different pgbench -c (num clients) and -s (scale) values. With this patch (and nothing else) on top of master (8f8b9be51fd7 to be precise): > fix-statistics-reporting-in-logical-replication-work.patch logical replication is still often failing (as expected, I suppose; it seems because of "inital snapshot too large") but indeed I do not see the 'TRAP: FailedAssertion in pgstat.c' anymore. (If there is any other configuration of patches worth testing please let me know) thanks Erik Rijkers
On Wed, May 03, 2017 at 01:23:19PM +0200, Erik Rijkers wrote: > On 2017-05-03 08:17, Petr Jelinek wrote: > >On 02/05/17 20:43, Robert Haas wrote: > >>On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > > >>>code path that calls CommitTransactionCommand() should have one, no? > >> > >>Is there anything left to be committed here? > >> > > > >Afaics the fix was not committed. Peter wanted more comprehensive fix > >which didn't happen. I think something like attached should do the job. > > I'm running my pgbench-over-logical-replication test in chunk of 15 minutes, > wth different pgbench -c (num clients) and -s (scale) values. > > With this patch (and nothing else) on top of master (8f8b9be51fd7 to be > precise): > > >fix-statistics-reporting-in-logical-replication-work.patch > > logical replication is still often failing (as expected, I suppose; it seems > because of "inital snapshot too large") but indeed I do not see the 'TRAP: > FailedAssertion in pgstat.c' anymore. > > (If there is any other configuration of patches worth testing please let me > know) [Action required within three days. This is a generic notification.] The above-described topic is currently a PostgreSQL 10 open item. Peter, since you committed the patch believed to have created it, you own this open item. If some other commit is more relevant or if this does not belong as a v10 open item, please let us know. Otherwise, please observe the policy on open item ownership[1] and send a status update within three calendar days of this message. Include a date for your subsequent status update. Testers may discover new open items at any time, and I want to plan to get them all fixed well in advance of shipping v10. Consequently, I will appreciate your efforts toward speedy resolution. Thanks. [1] https://www.postgresql.org/message-id/20170404140717.GA2675809%40tornado.leadboat.com
On 03/05/17 13:23, Erik Rijkers wrote: > On 2017-05-03 08:17, Petr Jelinek wrote: >> On 02/05/17 20:43, Robert Haas wrote: >>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut > >>>> code path that calls CommitTransactionCommand() should have one, no? >>> >>> Is there anything left to be committed here? >>> >> >> Afaics the fix was not committed. Peter wanted more comprehensive fix >> which didn't happen. I think something like attached should do the job. > > I'm running my pgbench-over-logical-replication test in chunk of 15 > minutes, wth different pgbench -c (num clients) and -s (scale) values. > > With this patch (and nothing else) on top of master (8f8b9be51fd7 to be > precise): > >> fix-statistics-reporting-in-logical-replication-work.patch > > logical replication is still often failing (as expected, I suppose; it > seems because of "inital snapshot too large") but indeed I do not see Yes that's different thing that we've been discussing a bit in snapbuild woes thread. > the 'TRAP: FailedAssertion in pgstat.c' anymore. > > (If there is any other configuration of patches worth testing please let > me know) > Thanks, so the patch works. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek <petr.jelinek@2ndquadrant.com> wrote: > On 03/05/17 13:23, Erik Rijkers wrote: >> On 2017-05-03 08:17, Petr Jelinek wrote: >>> On 02/05/17 20:43, Robert Haas wrote: >>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut >> >>>>> code path that calls CommitTransactionCommand() should have one, no? >>>> >>>> Is there anything left to be committed here? >>>> >>> >>> Afaics the fix was not committed. Peter wanted more comprehensive fix >>> which didn't happen. I think something like attached should do the job. >> >> I'm running my pgbench-over-logical-replication test in chunk of 15 >> minutes, wth different pgbench -c (num clients) and -s (scale) values. >> >> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be >> precise): >> >>> fix-statistics-reporting-in-logical-replication-work.patch >> >> logical replication is still often failing (as expected, I suppose; it >> seems because of "inital snapshot too large") but indeed I do not see > > Yes that's different thing that we've been discussing a bit in snapbuild > woes thread. > >> the 'TRAP: FailedAssertion in pgstat.c' anymore. >> >> (If there is any other configuration of patches worth testing please let >> me know) >> > > Thanks, so the patch works. > I think that we should commit the local transaction that did initial data copy, and then report stat as well. Currently table sync worker doesn't commit the local transaction in LogicalRepSyncTableStart (maybe until apply commit record?) if its status is changed to SUBREL_STATE_CATCHUP. That's why the table sync worker issues assertion failure. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center
On 5/5/17 07:13, Petr Jelinek wrote: > Yes that's different thing that we've been discussing a bit in snapbuild > woes thread. > >> the 'TRAP: FailedAssertion in pgstat.c' anymore. >> >> (If there is any other configuration of patches worth testing please let >> me know) > > Thanks, so the patch works. Committed, with s/xact_started/started_tx/, to match nearby code. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 08/05/17 17:52, Masahiko Sawada wrote: > On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek > <petr.jelinek@2ndquadrant.com> wrote: >> On 03/05/17 13:23, Erik Rijkers wrote: >>> On 2017-05-03 08:17, Petr Jelinek wrote: >>>> On 02/05/17 20:43, Robert Haas wrote: >>>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut >>> >>>>>> code path that calls CommitTransactionCommand() should have one, no? >>>>> >>>>> Is there anything left to be committed here? >>>>> >>>> >>>> Afaics the fix was not committed. Peter wanted more comprehensive fix >>>> which didn't happen. I think something like attached should do the job. >>> >>> I'm running my pgbench-over-logical-replication test in chunk of 15 >>> minutes, wth different pgbench -c (num clients) and -s (scale) values. >>> >>> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be >>> precise): >>> >>>> fix-statistics-reporting-in-logical-replication-work.patch >>> >>> logical replication is still often failing (as expected, I suppose; it >>> seems because of "inital snapshot too large") but indeed I do not see >> >> Yes that's different thing that we've been discussing a bit in snapbuild >> woes thread. >> >>> the 'TRAP: FailedAssertion in pgstat.c' anymore. >>> >>> (If there is any other configuration of patches worth testing please let >>> me know) >>> >> >> Thanks, so the patch works. >> > > I think that we should commit the local transaction that did initial > data copy, and then report stat as well. Currently table sync worker > doesn't commit the local transaction in LogicalRepSyncTableStart > (maybe until apply commit record?) if its status is changed to > SUBREL_STATE_CATCHUP. That's why the table sync worker issues > assertion failure. > That would fix the assert as well yes, but it would also mean that if the worker crashed between the initial copy and the end of catchup there would be no way to restart it without manual intervention from user since the synchronization position would be lost. Hence the fix I proposed which does it differently and has the whole sync in a single transaction. -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 08/05/17 18:12, Peter Eisentraut wrote: > On 5/5/17 07:13, Petr Jelinek wrote: >> Yes that's different thing that we've been discussing a bit in snapbuild >> woes thread. >> >>> the 'TRAP: FailedAssertion in pgstat.c' anymore. >>> >>> (If there is any other configuration of patches worth testing please let >>> me know) >> >> Thanks, so the patch works. > > Committed, with s/xact_started/started_tx/, to match nearby code. > Thanks! -- Petr Jelinek http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On Tue, May 9, 2017 at 1:26 AM, Petr Jelinek <petr.jelinek@2ndquadrant.com> wrote: > On 08/05/17 17:52, Masahiko Sawada wrote: >> On Fri, May 5, 2017 at 8:13 PM, Petr Jelinek >> <petr.jelinek@2ndquadrant.com> wrote: >>> On 03/05/17 13:23, Erik Rijkers wrote: >>>> On 2017-05-03 08:17, Petr Jelinek wrote: >>>>> On 02/05/17 20:43, Robert Haas wrote: >>>>>> On Thu, Apr 20, 2017 at 2:58 PM, Peter Eisentraut >>>> >>>>>>> code path that calls CommitTransactionCommand() should have one, no? >>>>>> >>>>>> Is there anything left to be committed here? >>>>>> >>>>> >>>>> Afaics the fix was not committed. Peter wanted more comprehensive fix >>>>> which didn't happen. I think something like attached should do the job. >>>> >>>> I'm running my pgbench-over-logical-replication test in chunk of 15 >>>> minutes, wth different pgbench -c (num clients) and -s (scale) values. >>>> >>>> With this patch (and nothing else) on top of master (8f8b9be51fd7 to be >>>> precise): >>>> >>>>> fix-statistics-reporting-in-logical-replication-work.patch >>>> >>>> logical replication is still often failing (as expected, I suppose; it >>>> seems because of "inital snapshot too large") but indeed I do not see >>> >>> Yes that's different thing that we've been discussing a bit in snapbuild >>> woes thread. >>> >>>> the 'TRAP: FailedAssertion in pgstat.c' anymore. >>>> >>>> (If there is any other configuration of patches worth testing please let >>>> me know) >>>> >>> >>> Thanks, so the patch works. >>> >> >> I think that we should commit the local transaction that did initial >> data copy, and then report stat as well. Currently table sync worker >> doesn't commit the local transaction in LogicalRepSyncTableStart >> (maybe until apply commit record?) if its status is changed to >> SUBREL_STATE_CATCHUP. That's why the table sync worker issues >> assertion failure. >> > > That would fix the assert as well yes, but it would also mean that if > the worker crashed between the initial copy and the end of catchup there > would be no way to restart it without manual intervention from user > since the synchronization position would be lost. Hence the fix I > proposed which does it differently and has the whole sync in a single > transaction. I understood that the data synchronization even including apply logical record after changed to SUBREL_STATE_CATCHUP should be done in a single transaction. Thank you for explanation. Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center