Thread: history file on replica and double switchover
Hello! I`ve noticed, that when running switchover replica to master and back to replica, new history file is streamed to replica, but not archived, which is not great, because it breaks PITR if archiving is running on replica. The fix looks trivial. Bash script to reproduce the problem and patch are attached. -- Grigory Smolkin Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
Attachment
On 27.08.2020 16:02, Grigory Smolkin wrote: > Hello! > > I`ve noticed, that when running switchover replica to master and back > to replica, new history file is streamed to replica, but not archived, > which is not great, because it breaks PITR if archiving is running on > replica. The fix looks trivial. > Bash script to reproduce the problem and patch are attached. > Thanks for the report. I agree that it looks like a bug. For some reason, patch failed to apply on current master, even though I don't see any difference in the code. I'll attach this thread to the next commitfest, so it doesn't get lost. -- Anastasia Lubennikova Postgres Professional: http://www.postgrespro.com The Russian Postgres Company
On 2020/09/04 8:29, Anastasia Lubennikova wrote: > On 27.08.2020 16:02, Grigory Smolkin wrote: >> Hello! >> >> I`ve noticed, that when running switchover replica to master and back to replica, new history file is streamed to replica,but not archived, >> which is not great, because it breaks PITR if archiving is running on replica. The fix looks trivial. >> Bash script to reproduce the problem and patch are attached. >> > Thanks for the report. I agree that it looks like a bug. +1 + /* Mark history file as ready for archiving */ + if (XLogArchiveMode != ARCHIVE_MODE_OFF) + XLogArchiveNotify(fname); I agree that history file should be archived in the standby when archive_mode=always. But why do we need to do when archive_mode=on? I'm just concerned about the case where the primary and standby have the shared archive area, and archive_mode is on. > For some reason, patch failed to apply on current master, even though I don't see any difference in the code. > I'll attach this thread to the next commitfest, so it doesn't get lost. Thanks! Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION
On 2020/09/04 13:53, Fujii Masao wrote: > > > On 2020/09/04 8:29, Anastasia Lubennikova wrote: >> On 27.08.2020 16:02, Grigory Smolkin wrote: >>> Hello! >>> >>> I`ve noticed, that when running switchover replica to master and back to replica, new history file is streamed to replica,but not archived, >>> which is not great, because it breaks PITR if archiving is running on replica. The fix looks trivial. >>> Bash script to reproduce the problem and patch are attached. >>> >> Thanks for the report. I agree that it looks like a bug. > > +1 > > + /* Mark history file as ready for archiving */ > + if (XLogArchiveMode != ARCHIVE_MODE_OFF) > + XLogArchiveNotify(fname); > > I agree that history file should be archived in the standby when > archive_mode=always. But why do we need to do when archive_mode=on? > I'm just concerned about the case where the primary and standby > have the shared archive area, and archive_mode is on. So I updated the patch so that walreceiver marks the streamed history file as ready for archiving only when archive_mode=always. Patch attached. Thought? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION
Attachment
Hi,
My understanding is that the "archiver" won't even start if
"archive_mode = on" has been set on a "replica". Therefore, either
(XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode !=
ARCHIVE_MODE_OFF) will produce the same result.
Please see how the "archiver" is started in
src/backend/postmaster/postmaster.c
5227 /*
5228 * Start the archiver if we're responsible for
(re-)archiving received
5229 * files.
5230 */
5231 Assert(PgArchPID == 0);
5232 if (XLogArchivingAlways())
5233 PgArchPID = pgarch_start();
I did run the nice script "double_switchover.sh" using either "always"
or "on" on patch v1 and v2. They all generate the same results below. In
other words, whether history file (00000003.history) is archived or not
depends on "archive_mode" settings.
echo "archive_mode = always" >> ${PGDATA_NODE2}/postgresql.auto.conf
$ ls -l archive
-rw------- 1 david david 16777216 Sep 24 14:40 000000010000000000000002
... ...
-rw------- 1 david david 16777216 Sep 24 14:40 00000002000000000000000A
-rw------- 1 david david 41 Sep 24 14:40 00000002.history
-rw------- 1 david david 83 Sep 24 14:40 00000003.history
echo "archive_mode = on" >> ${PGDATA_NODE2}/postgresql.auto.conf
$ ls -l archive
-rw------- 1 david david 16777216 Sep 24 14:47 000000010000000000000002
... ...
-rw------- 1 david david 16777216 Sep 24 14:47 00000002000000000000000A
-rw------- 1 david david 41 Sep 24 14:47 00000002.history
Personally, I prefer patch v2 since it allies to the statement here,
https://www.postgresql.org/docs/13/warm-standby.html#CONTINUOUS-ARCHIVING-IN-STANDBY
"If archive_mode is set to on, the archiver is not enabled during
recovery or standby mode. If the standby server is promoted, it will
start archiving after the promotion, but will not archive any WAL it did
not generate itself."
By the way, I think the last part of the sentence should be changed to
something like below:
"but will not archive any WAL, which was not generated by itself."
Best regards,
David
On 2020-09-17 10:18 a.m., Fujii Masao wrote:
>
>
> On 2020/09/04 13:53, Fujii Masao wrote:
>>
>>
>> On 2020/09/04 8:29, Anastasia Lubennikova wrote:
>>> On 27.08.2020 16:02, Grigory Smolkin wrote:
>>>> Hello!
>>>>
>>>> I`ve noticed, that when running switchover replica to master and
>>>> back to replica, new history file is streamed to replica, but not
>>>> archived,
>>>> which is not great, because it breaks PITR if archiving is running
>>>> on replica. The fix looks trivial.
>>>> Bash script to reproduce the problem and patch are attached.
>>>>
>>> Thanks for the report. I agree that it looks like a bug.
>>
>> +1
>>
>> + /* Mark history file as ready for archiving */
>> + if (XLogArchiveMode != ARCHIVE_MODE_OFF)
>> + XLogArchiveNotify(fname);
>>
>> I agree that history file should be archived in the standby when
>> archive_mode=always. But why do we need to do when archive_mode=on?
>> I'm just concerned about the case where the primary and standby
>> have the shared archive area, and archive_mode is on.
>
> So I updated the patch so that walreceiver marks the streamed history
> file
> as ready for archiving only when archive_mode=always. Patch attached.
> Thought?
>
> Regards,
>
--
David
Software Engineer
Highgo Software Inc. (Canada)
www.highgo.ca
On 2020/09/25 8:15, David Zhang wrote:
> Hi,
>
> My understanding is that the "archiver" won't even start if "archive_mode = on" has been set on a "replica".
Therefore,either (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode != ARCHIVE_MODE_OFF) will produce the
sameresult.
Yes, the archiver isn't invoked in the standby when archive_mode=on.
But, with the original patch, walreceiver creates .ready archive status file
even when archive_mode=on and no archiver is running. This causes
the history file to be archived after the standby is promoted and
the archiver is invoked.
With my patch, walreceiver creates .ready archive status for the history file
only when archive_mode=always, like it does for the streamed WAL files.
This is the difference between those two patches. To prevent the streamed
timeline history file from being archived, IMO we should use the condition
archive_mode==always in the walreceiver.
>
> Please see how the "archiver" is started in src/backend/postmaster/postmaster.c
>
> 5227 /*
> 5228 * Start the archiver if we're responsible for (re-)archiving received
> 5229 * files.
> 5230 */
> 5231 Assert(PgArchPID == 0);
> 5232 if (XLogArchivingAlways())
> 5233 PgArchPID = pgarch_start();
>
> I did run the nice script "double_switchover.sh" using either "always" or "on" on patch v1 and v2. They all generate
thesame results below. In other words, whether history file (00000003.history) is archived or not depends on
"archive_mode"settings.
>
> echo "archive_mode = always" >> ${PGDATA_NODE2}/postgresql.auto.conf
>
> $ ls -l archive
> -rw------- 1 david david 16777216 Sep 24 14:40 000000010000000000000002
> ... ...
> -rw------- 1 david david 16777216 Sep 24 14:40 00000002000000000000000A
> -rw------- 1 david david 41 Sep 24 14:40 00000002.history
> -rw------- 1 david david 83 Sep 24 14:40 00000003.history
>
> echo "archive_mode = on" >> ${PGDATA_NODE2}/postgresql.auto.conf
>
> $ ls -l archive
> -rw------- 1 david david 16777216 Sep 24 14:47 000000010000000000000002
> ... ...
> -rw------- 1 david david 16777216 Sep 24 14:47 00000002000000000000000A
> -rw------- 1 david david 41 Sep 24 14:47 00000002.history
>
>
> Personally, I prefer patch v2 since it allies to the statement here,
https://www.postgresql.org/docs/13/warm-standby.html#CONTINUOUS-ARCHIVING-IN-STANDBY
>
> "If archive_mode is set to on, the archiver is not enabled during recovery or standby mode. If the standby server is
promoted,it will start archiving after the promotion, but will not archive any WAL it did not generate itself."
>
> By the way, I think the last part of the sentence should be changed to something like below:
>
> "but will not archive any WAL, which was not generated by itself."
I'm not sure if this change is an improvement or not. But if we apply
the patch I proposed, maybe we should mention also history file here.
For example, "but will not archive any WAL or timeline history files
that it did not generate itself".
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
On 2020-09-24 5:00 p.m., Fujii Masao wrote:
>
>
> On 2020/09/25 8:15, David Zhang wrote:
>> Hi,
>>
>> My understanding is that the "archiver" won't even start if
>> "archive_mode = on" has been set on a "replica". Therefore, either
>> (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode !=
>> ARCHIVE_MODE_OFF) will produce the same result.
>
> Yes, the archiver isn't invoked in the standby when archive_mode=on.
> But, with the original patch, walreceiver creates .ready archive
> status file
> even when archive_mode=on and no archiver is running. This causes
> the history file to be archived after the standby is promoted and
> the archiver is invoked.
>
> With my patch, walreceiver creates .ready archive status for the
> history file
> only when archive_mode=always, like it does for the streamed WAL files.
> This is the difference between those two patches. To prevent the streamed
> timeline history file from being archived, IMO we should use the
> condition
> archive_mode==always in the walreceiver.
+1
>
>
>>
>> Please see how the "archiver" is started in
>> src/backend/postmaster/postmaster.c
>>
>> 5227 /*
>> 5228 * Start the archiver if we're responsible for
>> (re-)archiving received
>> 5229 * files.
>> 5230 */
>> 5231 Assert(PgArchPID == 0);
>> 5232 if (XLogArchivingAlways())
>> 5233 PgArchPID = pgarch_start();
>>
>> I did run the nice script "double_switchover.sh" using either
>> "always" or "on" on patch v1 and v2. They all generate the same
>> results below. In other words, whether history file
>> (00000003.history) is archived or not depends on "archive_mode"
>> settings.
>>
>> echo "archive_mode = always" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>
>> $ ls -l archive
>> -rw------- 1 david david 16777216 Sep 24 14:40 000000010000000000000002
>> ... ...
>> -rw------- 1 david david 16777216 Sep 24 14:40 00000002000000000000000A
>> -rw------- 1 david david 41 Sep 24 14:40 00000002.history
>> -rw------- 1 david david 83 Sep 24 14:40 00000003.history
>>
>> echo "archive_mode = on" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>
>> $ ls -l archive
>> -rw------- 1 david david 16777216 Sep 24 14:47 000000010000000000000002
>> ... ...
>> -rw------- 1 david david 16777216 Sep 24 14:47 00000002000000000000000A
>> -rw------- 1 david david 41 Sep 24 14:47 00000002.history
>>
>>
>> Personally, I prefer patch v2 since it allies to the statement here,
>> https://www.postgresql.org/docs/13/warm-standby.html#CONTINUOUS-ARCHIVING-IN-STANDBY
>>
>> "If archive_mode is set to on, the archiver is not enabled during
>> recovery or standby mode. If the standby server is promoted, it will
>> start archiving after the promotion, but will not archive any WAL it
>> did not generate itself."
>>
>> By the way, I think the last part of the sentence should be changed
>> to something like below:
>>
>> "but will not archive any WAL, which was not generated by itself."
>
> I'm not sure if this change is an improvement or not. But if we apply
> the patch I proposed, maybe we should mention also history file here.
> For example, "but will not archive any WAL or timeline history files
> that it did not generate itself".
make sense for me
>
> Regards,
>
--
David
Software Engineer
Highgo Software Inc. (Canada)
www.highgo.ca
On 2020/09/25 13:00, David Zhang wrote:
> On 2020-09-24 5:00 p.m., Fujii Masao wrote:
>>
>>
>> On 2020/09/25 8:15, David Zhang wrote:
>>> Hi,
>>>
>>> My understanding is that the "archiver" won't even start if "archive_mode = on" has been set on a "replica".
Therefore,either (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode != ARCHIVE_MODE_OFF) will produce the
sameresult.
>>
>> Yes, the archiver isn't invoked in the standby when archive_mode=on.
>> But, with the original patch, walreceiver creates .ready archive status file
>> even when archive_mode=on and no archiver is running. This causes
>> the history file to be archived after the standby is promoted and
>> the archiver is invoked.
>>
>> With my patch, walreceiver creates .ready archive status for the history file
>> only when archive_mode=always, like it does for the streamed WAL files.
>> This is the difference between those two patches. To prevent the streamed
>> timeline history file from being archived, IMO we should use the condition
>> archive_mode==always in the walreceiver.
> +1
>>
>>
>>>
>>> Please see how the "archiver" is started in src/backend/postmaster/postmaster.c
>>>
>>> 5227 /*
>>> 5228 * Start the archiver if we're responsible for (re-)archiving received
>>> 5229 * files.
>>> 5230 */
>>> 5231 Assert(PgArchPID == 0);
>>> 5232 if (XLogArchivingAlways())
>>> 5233 PgArchPID = pgarch_start();
>>>
>>> I did run the nice script "double_switchover.sh" using either "always" or "on" on patch v1 and v2. They all
generatethe same results below. In other words, whether history file (00000003.history) is archived or not depends on
"archive_mode"settings.
>>>
>>> echo "archive_mode = always" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>>
>>> $ ls -l archive
>>> -rw------- 1 david david 16777216 Sep 24 14:40 000000010000000000000002
>>> ... ...
>>> -rw------- 1 david david 16777216 Sep 24 14:40 00000002000000000000000A
>>> -rw------- 1 david david 41 Sep 24 14:40 00000002.history
>>> -rw------- 1 david david 83 Sep 24 14:40 00000003.history
>>>
>>> echo "archive_mode = on" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>>
>>> $ ls -l archive
>>> -rw------- 1 david david 16777216 Sep 24 14:47 000000010000000000000002
>>> ... ...
>>> -rw------- 1 david david 16777216 Sep 24 14:47 00000002000000000000000A
>>> -rw------- 1 david david 41 Sep 24 14:47 00000002.history
>>>
>>>
>>> Personally, I prefer patch v2 since it allies to the statement here,
https://www.postgresql.org/docs/13/warm-standby.html#CONTINUOUS-ARCHIVING-IN-STANDBY
>>>
>>> "If archive_mode is set to on, the archiver is not enabled during recovery or standby mode. If the standby server
ispromoted, it will start archiving after the promotion, but will not archive any WAL it did not generate itself."
>>>
>>> By the way, I think the last part of the sentence should be changed to something like below:
>>>
>>> "but will not archive any WAL, which was not generated by itself."
>>
>> I'm not sure if this change is an improvement or not. But if we apply
>> the patch I proposed, maybe we should mention also history file here.
>> For example, "but will not archive any WAL or timeline history files
>> that it did not generate itself".
> make sense for me
So I included this change in the patch. Patch attached.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Attachment
Fujii Masao, David Zhang, Anastasia Lubennikova, many thanks to you for
efforts with this patch!
Can I mark it as ready for committer?
On 9/25/20 10:07 AM, Fujii Masao wrote:
>
>
> On 2020/09/25 13:00, David Zhang wrote:
>> On 2020-09-24 5:00 p.m., Fujii Masao wrote:
>>>
>>>
>>> On 2020/09/25 8:15, David Zhang wrote:
>>>> Hi,
>>>>
>>>> My understanding is that the "archiver" won't even start if
>>>> "archive_mode = on" has been set on a "replica". Therefore, either
>>>> (XLogArchiveMode == ARCHIVE_MODE_ALWAYS) or (XLogArchiveMode !=
>>>> ARCHIVE_MODE_OFF) will produce the same result.
>>>
>>> Yes, the archiver isn't invoked in the standby when archive_mode=on.
>>> But, with the original patch, walreceiver creates .ready archive
>>> status file
>>> even when archive_mode=on and no archiver is running. This causes
>>> the history file to be archived after the standby is promoted and
>>> the archiver is invoked.
>>>
>>> With my patch, walreceiver creates .ready archive status for the
>>> history file
>>> only when archive_mode=always, like it does for the streamed WAL files.
>>> This is the difference between those two patches. To prevent the
>>> streamed
>>> timeline history file from being archived, IMO we should use the
>>> condition
>>> archive_mode==always in the walreceiver.
>> +1
>>>
>>>
>>>>
>>>> Please see how the "archiver" is started in
>>>> src/backend/postmaster/postmaster.c
>>>>
>>>> 5227 /*
>>>> 5228 * Start the archiver if we're responsible for
>>>> (re-)archiving received
>>>> 5229 * files.
>>>> 5230 */
>>>> 5231 Assert(PgArchPID == 0);
>>>> 5232 if (XLogArchivingAlways())
>>>> 5233 PgArchPID = pgarch_start();
>>>>
>>>> I did run the nice script "double_switchover.sh" using either
>>>> "always" or "on" on patch v1 and v2. They all generate the same
>>>> results below. In other words, whether history file
>>>> (00000003.history) is archived or not depends on "archive_mode"
>>>> settings.
>>>>
>>>> echo "archive_mode = always" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>>>
>>>> $ ls -l archive
>>>> -rw------- 1 david david 16777216 Sep 24 14:40
>>>> 000000010000000000000002
>>>> ... ...
>>>> -rw------- 1 david david 16777216 Sep 24 14:40
>>>> 00000002000000000000000A
>>>> -rw------- 1 david david 41 Sep 24 14:40 00000002.history
>>>> -rw------- 1 david david 83 Sep 24 14:40 00000003.history
>>>>
>>>> echo "archive_mode = on" >> ${PGDATA_NODE2}/postgresql.auto.conf
>>>>
>>>> $ ls -l archive
>>>> -rw------- 1 david david 16777216 Sep 24 14:47
>>>> 000000010000000000000002
>>>> ... ...
>>>> -rw------- 1 david david 16777216 Sep 24 14:47
>>>> 00000002000000000000000A
>>>> -rw------- 1 david david 41 Sep 24 14:47 00000002.history
>>>>
>>>>
>>>> Personally, I prefer patch v2 since it allies to the statement
>>>> here,
>>>> https://www.postgresql.org/docs/13/warm-standby.html#CONTINUOUS-ARCHIVING-IN-STANDBY
>>>>
>>>> "If archive_mode is set to on, the archiver is not enabled during
>>>> recovery or standby mode. If the standby server is promoted, it
>>>> will start archiving after the promotion, but will not archive any
>>>> WAL it did not generate itself."
>>>>
>>>> By the way, I think the last part of the sentence should be changed
>>>> to something like below:
>>>>
>>>> "but will not archive any WAL, which was not generated by itself."
>>>
>>> I'm not sure if this change is an improvement or not. But if we apply
>>> the patch I proposed, maybe we should mention also history file here.
>>> For example, "but will not archive any WAL or timeline history files
>>> that it did not generate itself".
>> make sense for me
>
> So I included this change in the patch. Patch attached.
>
> Regards,
>
--
Grigory Smolkin
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
On 2020/09/26 2:58, Grigory Smolkin wrote: > Fujii Masao, David Zhang, Anastasia Lubennikova, many thanks to you for efforts with this patch! > Can I mark it as ready for committer? Ok, but I attached the updated version of the patch. It's helpful if you review that. In the latest patch, I changed walreceiver so that it creates .done file for the streamed timeline history file when archive_modeis NOT "always". Walreceiver does the same thing for the streamed WAL files to prevent them from being archived later. Without this, the streamedWAL files can exist in pg_wal without any archive status files, and then they will be archived later accidentallybecause of lack of archive status. OTOH, timeline history files will not be archived later even without archive status files. So there is no strong reason tomake walreceiver create .doen file for the timeline history files. But at least for me it's strange to keep the file inpg_wal without archive status. So for now I'm just inclined to create .done files.... Thought? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION
Attachment
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, passed Spec compliant: tested, passed Documentation: tested, passed "make installcheck-world" test was performed on branch "REL_13_STABLE" with tag "REL_13_0". The regression failed was notcaused by the "history file" patch, since it has the same error on my environment even without any patch. So the failureis not related, in other words, the patch "history_replica_v4.patch" is good for me. Below is the failed message when tested with and without "history_replica_v4.patch". t/004_timeline_switch.pl ............. ok t/005_replay_delay.pl ................ ok t/006_logical_decoding.pl ............ # Looks like your test exited with 29 before it could output anything. t/006_logical_decoding.pl ............ Dubious, test returned 29 (wstat 7424, 0x1d00) Failed 14/14 subtests t/007_sync_rep.pl .................... ok t/008_fsm_truncation.pl .............. ok t/009_twophase.pl .................... ok t/010_logical_decoding_timelines.pl .. # Looks like your test exited with 29 before it could output anything. t/010_logical_decoding_timelines.pl .. Dubious, test returned 29 (wstat 7424, 0x1d00) Failed 13/13 subtests t/011_crash_recovery.pl .............. ok t/012_subtransactions.pl ............. ok t/013_crash_restart.pl ............... ok t/014_unlogged_reinit.pl ............. ok t/015_promotion_pages.pl ............. ok t/016_min_consistency.pl ............. ok t/017_shm.pl ......................... ok t/018_wal_optimize.pl ................ ok t/019_replslot_limit.pl .............. ok t/020_archive_status.pl .............. ok Test Summary Report ------------------- t/006_logical_decoding.pl (Wstat: 7424 Tests: 0 Failed: 0) Non-zero exit status: 29 Parse errors: Bad plan. You planned 14 tests but ran 0. t/010_logical_decoding_timelines.pl (Wstat: 7424 Tests: 0 Failed: 0) Non-zero exit status: 29 Parse errors: Bad plan. You planned 13 tests but ran 0. Files=20, Tests=202, 103 wallclock secs ( 0.18 usr 0.04 sys + 21.20 cusr 23.52 csys = 44.94 CPU) Result: FAIL make[2]: *** [installcheck] Error 1 make[2]: Leaving directory `/home/david/git/postgres/src/test/recovery' make[1]: *** [installcheck-recovery-recurse] Error 2 make[1]: Leaving directory `/home/david/git/postgres/src/test' make: *** [installcheck-world-src/test-recurse] Error 2
On 2020/09/26 5:34, David Zhang wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation: tested, passed > > "make installcheck-world" test was performed on branch "REL_13_STABLE" with tag "REL_13_0". The regression failed was notcaused by the "history file" patch, since it has the same error on my environment even without any patch. So the failureis not related, in other words, the patch "history_replica_v4.patch" is good for me. Thanks for the check! I pushed the patch. Thanks! > Below is the failed message when tested with and without "history_replica_v4.patch". BTW, I could not reproduce this issue in my env. I'm not sure why this failure happened in your env.... Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION