reduce size of logs stored by buildfarm - Mailing list pgsql-hackers

From Álvaro Herrera
Subject reduce size of logs stored by buildfarm
Date
Msg-id 202511251218.zfs4nu2qnh2m@alvherre.pgsql
Whole thread Raw
Responses Re: reduce size of logs stored by buildfarm
Re: reduce size of logs stored by buildfarm
List pgsql-hackers
Hi,

I just noticed that one regress log file for the pg_dump test is about
7MB long because it contains totally unnecessary dump files in the
regress log output.  This is because the tests run pg_dump of schema
pg_catalog without redirecting the output anywhere, which means it goes
to stdout, and then `prove` helpfully puts it in the regress log file.
This is completely pointless.  We can make the log file 1 MB instead by
adding a --file parameter to that command, as 0001 attached.


We also have a couple of tests that leave rather large server logs
behind that are perhaps not very useful.  I would propose to run those
servers with log_statement=off and avoid some of the bloat.  0002 does
that.

Another possible approach, instead of individually tweaking t/*.pl files
to add 'log_statements=none', would be to set it that way in
PostgreSQL::Test::Cluster globally.  Right now we have
log_statements=all there, but maybe that was not a great idea.


I also noticed that an oauth test file contains a couple MBs of a
gigantic string of just 'x'.  I suppose that will compress well (with
gzip at least, the 2 MB file becomes 8 kB).  Still, it's kinda
ridiculous and useless to bloat a 67 kB file to 2 MB that way.  The
problem here is that we print 'sending JSON response' and the verbatim
response, included the 1 MB of "x" as '_pad_'.  Can we supress this?
0003 does that by simply cutting the string short at 10k, which reduces
the size of the log from 2 MB to some 86 kB.  Maybe there are better
ways to deal with this?  Jacob?


With these three patches, we go from having 62 MB bytes in files with
"log" in their names under testrun/ (except those that have "catalog")
to 30 MB.  I think that's not a bad reduction.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
 Are you not unsure you want to delete Firefox?
       [Not unsure]     [Not not unsure]    [Cancel]
                   http://smylers.hates-software.com/2008/01/03/566e45b2.html

Attachment

pgsql-hackers by date:

Previous
From: Viktor Holmberg
Date:
Subject: Re: ON CONFLICT DO SELECT (take 3)
Next
From: 河田達也
Date:
Subject: Re: [Proposal] Adding TRIM_SPACE option to COPY