Thread: Can we include capturing logs of pgdata/pg_upgrade_output.d/*/log in buildfarm

Hi,

Recently there have been few upgrade tap test failures in buildfarm
like in [1] & [2]. Analysing these failures requires the log files
that are getting generated from src/bin/pg_upgrade at the following
locations:
tmp_check/*/pgdata/pg_upgrade_output.d/*/*.txt   - e.g.
tmp_check/t_004_subscription_new_sub1_data/pgdata/pg_upgrade_output.d/20240214T052229.045/subs_invalid.txt
tmp_check/*/pgdata/pg_upgrade_output.d/*/*/*.log - e.g.
tmp_check/t_004_subscription_new_sub1_data/pgdata/pg_upgrade_output.d/20240214T052229.045/log/pg_upgrade_server.log

First regex is the testname_clusterinstance_data, second regex is the
timestamp used for pg_upgrade, third regex is for the text files
generated by pg_upgrade and fourth regex is for the log files
generated by pg_upgrade.

Can we include these log files also in the buildfarm?

[1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
[2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20

Regards,
Vignesh

On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
> First regex is the testname_clusterinstance_data, second regex is the
> timestamp used for pg_upgrade, third regex is for the text files
> generated by pg_upgrade and fourth regex is for the log files
> generated by pg_upgrade.
>
> Can we include these log files also in the buildfarm?
>
> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20

Indeed, these lack some patterns.  Why not sending a pull request
around [1] to get more patterns covered?
[1]: https://github.com/PGBuildFarm/client-code/blob/main/PGBuild/Modules/TestUpgrade.pm
--
Michael

On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
>
> On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
> > First regex is the testname_clusterinstance_data, second regex is the
> > timestamp used for pg_upgrade, third regex is for the text files
> > generated by pg_upgrade and fourth regex is for the log files
> > generated by pg_upgrade.
> >
> > Can we include these log files also in the buildfarm?
> >
> > [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
> > [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
>
> Indeed, these lack some patterns.  Why not sending a pull request
> around [1] to get more patterns covered?

I have added a few more patterns to include the pg_upgrade generated
files. The attached patch has the changes for the same.
Adding Andrew also to get his thoughts on this.

Regards,
Vignesh

On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
> > > First regex is the testname_clusterinstance_data, second regex is the
> > > timestamp used for pg_upgrade, third regex is for the text files
> > > generated by pg_upgrade and fourth regex is for the log files
> > > generated by pg_upgrade.
> > >
> > > Can we include these log files also in the buildfarm?
> > >
> > > [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
> > > [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
> >
> > Indeed, these lack some patterns.  Why not sending a pull request
> > around [1] to get more patterns covered?
>
> I have added a few more patterns to include the pg_upgrade generated
> files. The attached patch has the changes for the same.
> Adding Andrew also to get his thoughts on this.

I have added the following commitfest entry for this:
https://commitfest.postgresql.org/47/4850/

Regards,
Vignesh

On 2024-02-25 Su 11:18, vignesh C wrote:
> On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
>> On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
>>> On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
>>>> First regex is the testname_clusterinstance_data, second regex is the
>>>> timestamp used for pg_upgrade, third regex is for the text files
>>>> generated by pg_upgrade and fourth regex is for the log files
>>>> generated by pg_upgrade.
>>>>
>>>> Can we include these log files also in the buildfarm?
>>>>
>>>> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
>>>> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
>>> Indeed, these lack some patterns.  Why not sending a pull request
>>> around [1] to get more patterns covered?
>> I have added a few more patterns to include the pg_upgrade generated
>> files. The attached patch has the changes for the same.
>> Adding Andrew also to get his thoughts on this.
> I have added the following commitfest entry for this:
> https://commitfest.postgresql.org/47/4850/
>

Buildfarm code patches do not belong in the Commitfest, I have marked 
the item as rejected. You can send me patches directly or add a PR to 
the buildfarm's github repo.

In this case the issue on drongo was a typo, the fix for which I had 
forgotten to propagate back in December. Note that the buildfarm's 
TestUpgrade.pm module is only used for branches < 15. For branches >= 15 
we run the standard TAP test and this module does nothing.

More generally, the collection of logs etc. for pg_upgrade will improve 
with the next release, which will be soon after I return from a vacation 
in about 2 weeks - experience shows that making releases just before a 
vacation is not a good idea :-)


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

On Mon, 26 Feb 2024 at 10:57, Andrew Dunstan <andrew@dunslane.net> wrote:
>
>
> On 2024-02-25 Su 11:18, vignesh C wrote:
> > On Thu, 15 Feb 2024 at 08:36, vignesh C <vignesh21@gmail.com> wrote:
> >> On Thu, 15 Feb 2024 at 07:24, Michael Paquier <michael@paquier.xyz> wrote:
> >>> On Wed, Feb 14, 2024 at 03:51:08PM +0530, vignesh C wrote:
> >>>> First regex is the testname_clusterinstance_data, second regex is the
> >>>> timestamp used for pg_upgrade, third regex is for the text files
> >>>> generated by pg_upgrade and fourth regex is for the log files
> >>>> generated by pg_upgrade.
> >>>>
> >>>> Can we include these log files also in the buildfarm?
> >>>>
> >>>> [1] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2024-02-10%2007%3A03%3A10
> >>>> [2] - https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=drongo&dt=2023-12-07%2003%3A56%3A20
> >>> Indeed, these lack some patterns.  Why not sending a pull request
> >>> around [1] to get more patterns covered?
> >> I have added a few more patterns to include the pg_upgrade generated
> >> files. The attached patch has the changes for the same.
> >> Adding Andrew also to get his thoughts on this.
> > I have added the following commitfest entry for this:
> > https://commitfest.postgresql.org/47/4850/
> >
>
> Buildfarm code patches do not belong in the Commitfest, I have marked
> the item as rejected. You can send me patches directly or add a PR to
> the buildfarm's github repo.

Ok, I will send over the patch directly for the required things.

>
> In this case the issue on drongo was a typo, the fix for which I had
> forgotten to propagate back in December. Note that the buildfarm's
> TestUpgrade.pm module is only used for branches < 15. For branches >= 15
> we run the standard TAP test and this module does nothing.
>
> More generally, the collection of logs etc. for pg_upgrade will improve
> with the next release, which will be soon after I return from a vacation
> in about 2 weeks - experience shows that making releases just before a
> vacation is not a good idea :-)

Thanks, that will be helpful.

Regards,
Vignesh