Thread: [HACKERS] Need a builtin way to run all tests faster manner
Hi, Right now check-world, a sensible thing to run before commits that aren't very narrow, takes a long time. To the point it seems to impact development velocity enough that it's more sensible to just skip it, and rely on the buildfarm. That's not a great solution. A large amount of the time is actually spent doing completely redundant initdb, cluster start/stop work, and running tests serially that could be run in parallel. A single check-world on my machine takes over 20min. That's just not realistic to run without hurting development pace. We can avoid a lot of redundant work (skip redundant initdb & cluster start/stop), and we can quite easily parallelize others. The problem is that doing so isn't something entirely trivially scriptable, e.g. make installcheck-world doesn't run all tests, and it doesn't do so in parallel. Scripting it locally also has the issue that it's very easy to not notice new stuff being added by others. As an example of the speedups, here's the comparison for contrib: make -C contrib: 2m21.056s make -C contrib installcheck 0m30.672s make -C contrib -j16 -s -Otarget installcheck USE_MODULE_DB=1 0m10.418s that's not an entirely fair comparison however, because test_decoding doesn't to installcheck, but the general principle holds. This is obviously a large difference. A lot of the slow tap tests could be run serially, recoverying a lot more time. There's also some tests simply taking way too long, e.g. the pg_dump tests just do largely redundant tests for 30s. I'm not quite sure what the best way to attack this is, but I think we need to do something. Greetings, Andres Freund
Andres, * Andres Freund (andres@anarazel.de) wrote: > There's also some tests simply taking way too long, e.g. the pg_dump > tests just do largely redundant tests for 30s. About half of the time in the pg_dump case, at least, appears to be in 010_dump_connstr.pl. I've not looked into it, but based on the test names it does look like some of those tests might be redundant to what is already being covered in 002_pg_dump.pl. Of course, the way the TAP tests run, they require an initdb and startup of the postmaster, and that's a somewhat fixed amount of time that any TAP test is going to take. I'm all for improving things certainly, though, well, while the pg_dump tests do a lot, they also try to cover a good bit of the code in pg_dump.c which is quite large. I wouldn't want to reduce our code coverage just to make the regression tests go faster. The changes to the pg_dump tests that I'm hoping to push soon will at least reduce the output some, if not the length of time taken, while providing more code coverage. All that said, 30s out of 20m (which seems to more-or-less match what I get locally too) makes me really wonder if that's the place that we need to focus. Is it really the longest running test we have and it's just that we have tons of them that are doing initdb over and over again? > I'm not quite sure what the best way to attack this is, but I think we > need to do something. I tend to agree with this, though I haven't got any great answers, unfortunately. Thanks! Stephen
Stephen Frost <sfrost@snowman.net> writes:
> * Andres Freund (andres@anarazel.de) wrote:
>> I'm not quite sure what the best way to attack this is, but I think we
>> need to do something.
> I tend to agree with this, though I haven't got any great answers,
> unfortunately.
I don't want to reduce test coverage either. I think the most painless
way to improve matters would just be to work harder on running tests in
parallel. I think most devs these days do most of their work on 4- or
8-core machines, yet almost everything except the core regression tests
is depressingly serial. I think we could likely get a 2x or better
reduction in total runtime without too much work just by attacking that.
regards, tom lane
On 2017-03-06 19:45:27 -0500, Tom Lane wrote: > Stephen Frost <sfrost@snowman.net> writes: > > * Andres Freund (andres@anarazel.de) wrote: > >> I'm not quite sure what the best way to attack this is, but I think we > >> need to do something. > > > I tend to agree with this, though I haven't got any great answers, > > unfortunately. > > I don't want to reduce test coverage either. I think the most painless > way to improve matters would just be to work harder on running tests in > parallel. I think most devs these days do most of their work on 4- or > 8-core machines, yet almost everything except the core regression tests > is depressingly serial. I think we could likely get a 2x or better > reduction in total runtime without too much work just by attacking that. A lot more probably, based on my preliminary tests / my local test script. I'm just not quite sure what the best way is to make it easier to run tests in parallel within the tree. The best I can come up so far is a toplevel target that creates the temp install, starts a cluster and then runs the 'installcheck-or-check' target on all the subdirectories via recursion. Individual makefiles can either use the pre-existing cluster (most of of contrib for example), or use the temporary install and run their pre-existing check target using that (the tap tests, test_decoding, ...). Requires editing a bunch of Makefiles to take advantage. But I don't really see anything that doesn't require that. - Andres
FWIW, +1 on improving matters here. Andres Freund wrote: > The best I can come up so far is a toplevel target that creates the temp > install, starts a cluster and then runs the 'installcheck-or-check' > target on all the subdirectories via recursion. Individual makefiles can > either use the pre-existing cluster (most of of contrib for example), or > use the temporary install and run their pre-existing check target using > that (the tap tests, test_decoding, ...). I think a toplevel installcheck-or-check target is a good first step (though definitely lets find a better name). Just being able to run all tests without the need for 95% of pointless initdb's would be helpful enough. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > FWIW, +1 on improving matters here. +1 also. I don't see what's wrong with relying on buildfarm though; testing is exactly what its there for. If we had a two-stage process, where committers can issue "trial commits" as a way of seeing if the build farm likes things. If they do, we can push to the main repo. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: [HACKERS] Need a builtin way to run all tests faster manner
From
ilmari@ilmari.org (Dagfinn Ilmari Mannsåker)
Date:
Simon Riggs <simon@2ndquadrant.com> writes: > On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > >> FWIW, +1 on improving matters here. > > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we had a two-stage process, where committers can issue "trial > commits" as a way of seeing if the build farm likes things. If they > do, we can push to the main repo. In perl we do this by having the smoke testers (build farm) pick up branches with a specific prefix (smoke-me/ in our case, typically smoke-me/<username>/<topic>), in addition to the blead (master) and maint-x.y (release) branches. -- "I use RMS as a guide in the same way that a boat captain would usea lighthouse. It's good to know where it is, but yougenerallydon't want to find yourself in the same spot." - Tollef Fog Heen
On 03/07/2017 07:58 AM, Simon Riggs wrote: > On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > >> FWIW, +1 on improving matters here. > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we had a two-stage process, where committers can issue "trial > commits" as a way of seeing if the build farm likes things. If they > do, we can push to the main repo. > I'm happy to work on this. Not quite sure how it would work, but I'm open to any suggestions. cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On Tue, Mar 7, 2017 at 7:24 AM, Andrew Dunstan <andrew.dunstan@2ndquadrant.com> wrote:
On 03/07/2017 07:58 AM, Simon Riggs wrote:
> On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote:
>
>> FWIW, +1 on improving matters here.
> +1 also.
>
> I don't see what's wrong with relying on buildfarm though; testing is
> exactly what its there for.
>
> If we had a two-stage process, where committers can issue "trial
> commits" as a way of seeing if the build farm likes things. If they
> do, we can push to the main repo.
>
I'm happy to work on this. Not quite sure how it would work, but I'm
open to any suggestions.
Assuming the intention is a service for *committers only*, I suggest setting up a separate (closed) git repository that committers can push to and a separate set of BF animals could work from. We could just have it do all branches in it, no need to filter that way as long as we keep it a separate repo.
There have also on and off been discussions about building arbitrary patches as they are sent to the mailinglists. Doing that without any committer (or other trusted party) as a filter is a completely different challenge of course, given that it basically amounts to downloading and running random code off the internet.
But doing just the first would make it a lot easier, and probably still be of good value.
An in-between could be to hook something off the CF app, but one important question is how important covering many platforms is. Since we already have good functionality for doing that in the buildfarm, it makes sense to utilize that if we can.
On 2017-03-07 20:58:35 +0800, Simon Riggs wrote: > On 7 March 2017 at 20:36, Alvaro Herrera <alvherre@2ndquadrant.com> wrote: > > > FWIW, +1 on improving matters here. > > +1 also. > > I don't see what's wrong with relying on buildfarm though; testing is > exactly what its there for. > > If we had a two-stage process, where committers can issue "trial > commits" as a way of seeing if the build farm likes things. If they > do, we can push to the main repo. Personally that's not addressing my main concern, which is that the latency of getting done with some patch/topic takes a long while. If I have to wait for the buildfarm to check some preliminary patch, I still have to afterwards work on pushing it to master. And very likely my local check would finish a lot faster than a bunch of buildfarm animals - I have after all a plenty powerfull machine, lots of cores, fast ssd, lots of memory, ... So I really want faster end-to-end test, not less cpu time spent on my own machine. - Andres
On Tue, Mar 7, 2017 at 11:23 PM, Andres Freund <andres@anarazel.de> wrote: > Personally that's not addressing my main concern, which is that the > latency of getting done with some patch/topic takes a long while. If I > have to wait for the buildfarm to check some preliminary patch, I still > have to afterwards work on pushing it to master. And very likely my > local check would finish a lot faster than a bunch of buildfarm animals > - I have after all a plenty powerfull machine, lots of cores, fast ssd, > lots of memory, ... > > So I really want faster end-to-end test, not less cpu time spent on my > own machine. Yeah. I think the buildfarm-for-test-commits or maybe buildfarm-for-approved-branches-belonging-to-people-we-basically-trust idea isn't a bad one, but it's not a substitute for $SUBJECT. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
On 3/6/17 19:53, Andres Freund wrote: > I'm just not quite sure what the best way is to make it easier to run > tests in parallel within the tree. make check-world -j2 seems to run fine for me. With higher -j I appear to be running out of memory or disks space, so I haven't checked that any further, but it seems possible. You can also run prove with a -j option. And we could parallelize some of the contrib/pl tests, e.g., plpython. (The problem is that parallel make and parallel tests together might explode a bit, so we might want some way to control which aspect we parallelize.) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote: > On 3/6/17 19:53, Andres Freund wrote: > > I'm just not quite sure what the best way is to make it easier to run > > tests in parallel within the tree. > > make check-world -j2 seems to run fine for me. Hm, I at least used to get a lot of spurious failures with this. I e.g. don't think the free port selection is race free. Also that doesn't solve the issue of the time spent in repetitive initdbs - but that's mainly in contrib, and we probably solve that there by having a special target in contrib/ building a cluster once. > You can also run prove with a -j option. Ah, interesting. > (The problem is that parallel make and parallel tests together might > explode a bit, so we might want some way to control which aspect we > parallelize.) Yea, I indeed see that. Probably could do a top-level prerequisite for check on a check-world subset that includes docs? - Andres
On Thu, Mar 9, 2017 at 6:49 AM, Andres Freund <andres@anarazel.de> wrote: > On 2017-03-08 16:32:49 -0500, Peter Eisentraut wrote: >> On 3/6/17 19:53, Andres Freund wrote: >> > I'm just not quite sure what the best way is to make it easier to run >> > tests in parallel within the tree. >> >> make check-world -j2 seems to run fine for me. > > Hm, I at least used to get a lot of spurious failures with this. I > e.g. don't think the free port selection is race free. pg_regress is running each Postgres instance in a separate Unix socket directory, so the port selection is not problem. PostgresNode.pm is also careful about that. > Also that > doesn't solve the issue of the time spent in repetitive initdbs - but > that's mainly in contrib, and we probably solve that there by having a > special target in contrib/ building a cluster once. Yeah, you can count the growing src/test/modules in that as well. -- Michael
On 3/8/17 16:49, Andres Freund wrote: >> make check-world -j2 seems to run fine for me. > > Hm, I at least used to get a lot of spurious failures with this. I > e.g. don't think the free port selection is race free. I was also not sure about that, but as Michael has pointed out, that doesn't matter anymore, because it now uses a private socket directory. I have been pounding it a bit, and every so often the test_decoding tests fail in mysterious ways, but otherwise it seems to work fine. I'm curious what you are seeing. Combining make -j10 and prove -j4, I get the run time down to 2 minutes and a bit, from 20+ minutes. Using the -O option if you have GNU make >=4 is also useful to get some more sane output. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 3/7/17 9:52 PM, Magnus Hagander wrote: > There have also on and off been discussions about building arbitrary > patches as they are sent to the mailinglists. Doing that without any > committer (or other trusted party) as a filter is a completely different > challenge of course, given that it basically amounts to downloading and > running random code off the internet. Perhaps https://travis-ci.org/ or something similar could be used for this. That avoids any issues about random code. -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On Fri, Mar 10, 2017 at 3:27 AM, Jim Nasby <jim.nasby@openscg.com> wrote:
On 3/7/17 9:52 PM, Magnus Hagander wrote:There have also on and off been discussions about building arbitrary
patches as they are sent to the mailinglists. Doing that without any
committer (or other trusted party) as a filter is a completely different
challenge of course, given that it basically amounts to downloading and
running random code off the internet.
Perhaps https://travis-ci.org/ or something similar could be used for this. That avoids any issues about random code.
AFAIK travis-ci would require us to use github as our hoster for all those things, and embrace that workflow, they don't support anything else.
There might be others that do, just not travis.
Re: [HACKERS] Need a builtin way to run all tests faster manner
From
ilmari@ilmari.org (Dagfinn Ilmari Mannsåker)
Date:
Magnus Hagander <magnus@hagander.net> writes: > AFAIK travis-ci would require us to use github as our hoster for all those > things, and embrace that workflow, they don't support anything else. > > There might be others that do, just not travis. It merely requires the repository to exist on GitHub, and postgresql.git is already mirrored to https://github.com/postgres/postgres. If there was a .travis.yml in the repo, people who fork it could easily enable Travis-CI for their fork, even the official repo isn't hooked up. - ilmari -- - Twitter seems more influential [than blogs] in the 'gets reported in the mainstream press' sense at least. - Matt McLeod - That'd be because the content of a tweet is easier to condense down to a mainstream media article. - Calle Dybedahl
On Fri, Mar 10, 2017 at 11:29 AM, Dagfinn Ilmari Mannsåker <ilmari@ilmari.org> wrote:
Magnus Hagander <magnus@hagander.net> writes:
> AFAIK travis-ci would require us to use github as our hoster for all those
> things, and embrace that workflow, they don't support anything else.
>
> There might be others that do, just not travis.
It merely requires the repository to exist on GitHub, and postgresql.git
is already mirrored to https://github.com/postgres/postgres. If there
was a .travis.yml in the repo, people who fork it could easily enable
Travis-CI for their fork, even the official repo isn't hooked up.
That's true. It would require a bunch of additional branches to make it useful in core though, but it śeems like it could be a useful thing for people to enable in their own personal forks/branches if they use github for their largest feature development. Could be a good idea to for example ave an example yml file somewhere on the wiki that people could put into their branches.
On 3/10/17 13:02, Magnus Hagander wrote: > for their largest feature development. Could be a good idea to for > example ave an example yml file somewhere on the wiki that people could > put into their branches. https://github.com/petere/postgresql/blob/travis/.travis.yml -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 3/10/17 03:27, Jim Nasby wrote: > Perhaps https://travis-ci.org/ or something similar could be used for > this. That avoids any issues about random code. That doesn't achieve any platform coverage, which is the main point here. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-03-10 01:59:53 -0500, Peter Eisentraut wrote: > On 3/8/17 16:49, Andres Freund wrote: > >> make check-world -j2 seems to run fine for me. > > > > Hm, I at least used to get a lot of spurious failures with this. I > > e.g. don't think the free port selection is race free. > > I was also not sure about that, but as Michael has pointed out, that > doesn't matter anymore, because it now uses a private socket directory. Yea, I had forgotten about that bit. > I have been pounding it a bit, and every so often the test_decoding > tests fail in mysterious ways, but otherwise it seems to work fine. I'm > curious what you are seeing. I do get regular issues, although the happen to not end up in visible failures. All the tests output their regression.diffs into the same place - which means there'll every now be a failure to remove the file, and if there were an actual failure it'd possibly end up being attributed to the wrong test. So I think we need to relocate that file to be relative to the sql/ | specs/ | expected/ directories? Curious about the test decoding thing - hadn't seen that here. Greetings, Andres Freund
On 3/10/17 1:09 PM, Peter Eisentraut wrote: > On 3/10/17 03:27, Jim Nasby wrote: >> Perhaps https://travis-ci.org/ or something similar could be used for >> this. That avoids any issues about random code. > > That doesn't achieve any platform coverage, which is the main point here. I don't think platform coverage is the first thing to worry about with patches, nor with ongoing development. The biggest win we'd get from something like Travis would be if the commitfest monitored for new patch files coming in for monitored threads and it created a new branch, applied the patches, and if they applied without error commit the branch and push to let Travis do it's thing. We wouldn't want that running in the main git repo, but it should be fine in a fork that's dedicated to that purpose. If the travis build failed, commitfest could notify the author. It could also rebase master into each branch on a daily basis so authors would know very quickly if something got committed that broke their patch. Obviously that doesn't remove the need for manual testing or the buildfarm, but it would at least let everyone know that the patch passed a smoke test. -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On Fri, Mar 10, 2017 at 2:53 PM, Jim Nasby <jim.nasby@openscg.com> wrote:
On 3/10/17 1:09 PM, Peter Eisentraut wrote:On 3/10/17 03:27, Jim Nasby wrote:Perhaps https://travis-ci.org/ or something similar could be used for
this. That avoids any issues about random code.
That doesn't achieve any platform coverage, which is the main point here.
I don't think platform coverage is the first thing to worry about with patches, nor with ongoing development.
I think we are talking about solving two different problems...
The biggest win we'd get from something like Travis would be if the commitfest monitored for new patch files coming in for monitored threads and it created a new branch, applied the patches, and if they applied without error commit the branch and push to let Travis do it's thing. We wouldn't want that running in the main git repo, but it should be fine in a fork that's dedicated to that purpose.
Travis specifically would not help us with this, due to the dependency on gifhub, but something that knows how to run "patch ... && configure && make && make check" in a container would.
I'm unsure what would be easiest -- have something drive a "throwaway github repo" off the data in the CF app and try to pull things from there, or to just spawn containers and run it directly without travis.
The bigger issue with those is the usual -- how do you handle patches that have dependencies on each other,because they're always going to show up as broken individually. I guess we could tell people doing those to just push a git branch on github and register that one in the CF app (which does have some very basic support for tracking that, but I doubt anybody uses it today).
If the travis build failed, commitfest could notify the author.
It could also rebase master into each branch on a daily basis so authors would know very quickly if something got committed that broke their patch.
It could at least verify that the patch still applies, yes.
Obviously that doesn't remove the need for manual testing or the buildfarm, but it would at least let everyone know that the patch passed a smoke test
It's definitely something that would be useful, but as you say, it solves a different problem.
//Magnus
On 3/10/17 2:05 PM, Magnus Hagander wrote: > > Travis specifically would not help us with this, due to the dependency > on gifhub, but something that knows how to run "patch ... && configure > && make && make check" in a container would. Who's updating https://github.com/postgres/postgres/ right now? Presumably that script would be the basis for this... > I'm unsure what would be easiest -- have something drive a "throwaway > github repo" off the data in the CF app and try to pull things from > there, or to just spawn containers and run it directly without travis. I'd be a bit nervous about creating our own container solution and opening that to automatically deploying patches. Travis (and other tools) already have that problem solved (or at least if they get hacked it's on them to clean up and not us :) Plus it'd be a heck of a lot more work on our side to set all that stuff up. > The bigger issue with those is the usual -- how do you handle patches > that have dependencies on each other,because they're always going to > show up as broken individually. I guess we could tell people doing those > to just push a git branch on github and register that one in the CF app > (which does have some very basic support for tracking that, but I doubt > anybody uses it today). If people use git format-patch it should JustWork(tm). Specifying a specific repo is another option. Even if we can't make it work for really complicated patches it might still be a win. > If the travis build failed, commitfest could notify the author. > > It could also rebase master into each branch on a daily basis so > authors would know very quickly if something got committed that > broke their patch. > > > It could at least verify that the patch still applies, yes. If the rebase was pushed to github and travis was setup, travis would then test the changes as well. -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On Fri, Mar 10, 2017 at 3:11 PM, Jim Nasby <jim.nasby@openscg.com> wrote:
On 3/10/17 2:05 PM, Magnus Hagander wrote:
Travis specifically would not help us with this, due to the dependency
on gifhub, but something that knows how to run "patch ... && configure
&& make && make check" in a container would.
Who's updating https://github.com/postgres/postgres/ right now? Presumably that script would be the basis for this...
Yes. It's a pure mirror script.
I'm pretty sure we *don't* want to push a branch for every single patch that goes in a CF all into our master repository...
We could use a completely separate repo on github for it if we want, yes. Then we just need to figure out how to get the patches there..
I'm unsure what would be easiest -- have something drive a "throwaway
github repo" off the data in the CF app and try to pull things from
there, or to just spawn containers and run it directly without travis.
I'd be a bit nervous about creating our own container solution and opening that to automatically deploying patches. Travis (and other tools) already have that problem solved (or at least if they get hacked it's on them to clean up and not us :)
Plus it'd be a heck of a lot more work on our side to set all that stuff up.
This is what I'm not so sure about. Setting up an empty container provided we only ever need to test the one thing and only ever need the one platform is not particularly complicated.
But if you can put together something that picks up the individual patches out of the mail threads in the CF app and keeps branch-tips in a git repo up-to-date with those, including feeding the results back into the app, then go for it :)
The bigger issue with those is the usual -- how do you handle patches
that have dependencies on each other,because they're always going to
show up as broken individually. I guess we could tell people doing those
to just push a git branch on github and register that one in the CF app
(which does have some very basic support for tracking that, but I doubt
anybody uses it today).
If people use git format-patch it should JustWork(tm). Specifying a specific repo is another option.
Right. But people don't use that all the time, and it's not currently a requirement on patch submitters. and we've traditionally been of the opinion that we don't want to put too much requirements on such things for submitters.
Even if we can't make it work for really complicated patches it might still be a win.
Definitely as long as we can keep some manual process for thos epatches.
If the travis build failed, commitfest could notify the author.
It could also rebase master into each branch on a daily basis so
authors would know very quickly if something got committed that
broke their patch.
It could at least verify that the patch still applies, yes.
If the rebase was pushed to github and travis was setup, travis would then test the changes as well.
Right.
//Magnus
On 3/10/17 2:18 PM, Magnus Hagander wrote: > But if you can put together something that picks up the individual > patches out of the mail threads in the CF app and keeps branch-tips in a > git repo up-to-date with those, including feeding the results back into > the app, then go for it :) Seems like an ideal project for someone not on -hackers... do we have a list of "How you can help Postgres besides hacking database code" anywhere? -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On 3/10/17 14:53, Jim Nasby wrote: > The biggest win we'd get from something like Travis would be if the > commitfest monitored for new patch files coming in for monitored threads > and it created a new branch, applied the patches, and if they applied > without error commit the branch and push to let Travis do it's thing. We > wouldn't want that running in the main git repo, but it should be fine > in a fork that's dedicated to that purpose. This has been discussed several times before, e.g., https://www.postgresql.org/message-id/54DD2413.8030201@gmx.net -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 3/10/17 5:57 PM, Peter Eisentraut wrote: > On 3/10/17 14:53, Jim Nasby wrote: >> The biggest win we'd get from something like Travis would be if the >> commitfest monitored for new patch files coming in for monitored threads >> and it created a new branch, applied the patches, and if they applied >> without error commit the branch and push to let Travis do it's thing. We >> wouldn't want that running in the main git repo, but it should be fine >> in a fork that's dedicated to that purpose. > > This has been discussed several times before, e.g., > > https://www.postgresql.org/message-id/54DD2413.8030201@gmx.net Maybe instead of having the commitfest app try and divine patches from the list it should be able to send patches to the list from a specified git repo/branch. Anyone that provides that info would have tests run automagically, patches sent, etc. Anyone who doesn't can just keep using the old process. -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On 3/10/17 14:43, Andres Freund wrote: > I do get regular issues, although the happen to not end up in visible > failures. All the tests output their regression.diffs into the same > place - which means there'll every now be a failure to remove the file, > and if there were an actual failure it'd possibly end up being > attributed to the wrong test. So I think we need to relocate that file > to be relative to the sql/ | specs/ | expected/ directories? I'm confused here. Every test suite runs in a separate directory, and for the tests in the same suite pg_regress will take care of running the tests in parallel and producing the diffs. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 3/10/17 19:00, Jim Nasby wrote: > Maybe instead of having the commitfest app try and divine patches from > the list it should be able to send patches to the list from a specified > git repo/branch. Anyone that provides that info would have tests run > automagically, patches sent, etc. Anyone who doesn't can just keep using > the old process. Those people who know what they're doing will presumably run all those checks before they submit a patch. It's those people who send in patches that don't apply cleanly or fail the tests that would benefit from this system. But if they're that careless, then they also won't take care to use this particular system correctly. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
>>>> make check-world -j2 seems to run fine for me.
> I have been pounding it a bit, and every so often the test_decoding
> tests fail in mysterious ways, but otherwise it seems to work fine. I'm
> curious what you are seeing.
For me, it's quite unreliable at make parallelism above -j2 or so.
I believe the core problem is that contrib/test_decoding's regresscheck
and isolationcheck targets each want to use ./tmp_check as their
--temp-instance. make has no reason to believe it shouldn't run those
two sub-jobs in parallel, but when it does, we get two postmasters trying
to share the same directory. This looks reasonably straightforward to
solve, but I'm not entirely familiar with the code here, and am not
sure what is the least ugly way to fix it.
regards, tom lane
I wrote:
> I believe the core problem is that contrib/test_decoding's regresscheck
> and isolationcheck targets each want to use ./tmp_check as their
> --temp-instance. make has no reason to believe it shouldn't run those
> two sub-jobs in parallel, but when it does, we get two postmasters trying
> to share the same directory. This looks reasonably straightforward to
> solve, but I'm not entirely familiar with the code here, and am not
> sure what is the least ugly way to fix it.
Enlarging on that: if I cd into contrib/test_decoding and do
"make check -j4" or so, it reliably fails. With the attached patch,
it passes. This is a localized patch that only fixes things for
contrib/test_decoding; what I'm wondering is if it would be better to
establish a more widespread convention that $(pg_isolation_regress_check)
should use a different --temp-instance directory than $(pg_regress_check)
does.
regards, tom lane
diff --git a/contrib/test_decoding/.gitignore b/contrib/test_decoding/.gitignore
index 1f95503..09d60bf 100644
*** a/contrib/test_decoding/.gitignore
--- b/contrib/test_decoding/.gitignore
***************
*** 3,5 ****
--- 3,6 ----
/isolation_output/
/regression_output/
/tmp_check/
+ /tmp_check_iso/
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index d2bc8b8..ea31d88 100644
*** a/contrib/test_decoding/Makefile
--- b/contrib/test_decoding/Makefile
*************** PGFILEDESC = "test_decoding - example of
*** 5,11 ****
# Note: because we don't tell the Makefile there are any regression tests,
# we have to clean those result files explicitly
! EXTRA_CLEAN = $(pg_regress_clean_files) ./regression_output ./isolation_output
ifdef USE_PGXS
PG_CONFIG = pg_config
--- 5,12 ----
# Note: because we don't tell the Makefile there are any regression tests,
# we have to clean those result files explicitly
! EXTRA_CLEAN = $(pg_regress_clean_files) ./regression_output \
! ./isolation_output ./tmp_check_iso
ifdef USE_PGXS
PG_CONFIG = pg_config
*************** isolationcheck: | submake-isolation subm
*** 59,64 ****
--- 60,66 ----
$(MKDIR_P) isolation_output
$(pg_isolation_regress_check) \
--temp-config $(top_srcdir)/contrib/test_decoding/logical.conf \
+ --temp-instance=./tmp_check_iso \
--outputdir=./isolation_output \
$(ISOLATIONCHECKS)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 3/10/17 6:06 PM, Peter Eisentraut wrote: > On 3/10/17 19:00, Jim Nasby wrote: >> Maybe instead of having the commitfest app try and divine patches from >> the list it should be able to send patches to the list from a specified >> git repo/branch. Anyone that provides that info would have tests run >> automagically, patches sent, etc. Anyone who doesn't can just keep using >> the old process. > > Those people who know what they're doing will presumably run all those > checks before they submit a patch. It's those people who send in > patches that don't apply cleanly or fail the tests that would benefit > from this system. But if they're that careless, then they also won't > take care to use this particular system correctly. It's actually a lot harder to mess up providing a git repo link than manually submitting patches to the mailing list. For most patches, it's also a hell of a lot faster to just submit a repo URL rather than dealing with patch files. Having this also means that reviewers can focus more on what the patch is actually doing instead of mechanical crap best left to a machine. Of course, *you* work on changes that are far more complex than any newbie will, and it wouldn't surprise me if such a feature wouldn't help you or other senior hackers at all. But AFAICT it wouldn't get in your way either. It would remove yet another burden for new hackers. Anyway, this is well off topic for the original thread... -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On 2017-03-11 12:05:23 -0500, Tom Lane wrote: > I wrote: > > I believe the core problem is that contrib/test_decoding's regresscheck > > and isolationcheck targets each want to use ./tmp_check as their > > --temp-instance. make has no reason to believe it shouldn't run those > > two sub-jobs in parallel, but when it does, we get two postmasters trying > > to share the same directory. This looks reasonably straightforward to > > solve, but I'm not entirely familiar with the code here, and am not > > sure what is the least ugly way to fix it. > > Enlarging on that: if I cd into contrib/test_decoding and do > "make check -j4" or so, it reliably fails. Yep, can reproduce here as well. Interesting that, with -j16, I could survive several dozen runs from the toplevel locally. > This is a localized patch that only fixes things for > contrib/test_decoding; what I'm wondering is if it would be better to > establish a more widespread convention that > $(pg_isolation_regress_check) should use a different --temp-instance > directory than $(pg_regress_check) does. I think that'd be a good plan. We probably should also keep --outputdir seperate (which test_decoding/Makefile does, but pg_isolation_regress_check doesn't)? - Andres
Jim Nasby <jim.nasby@openscg.com> writes:
> It's actually a lot harder to mess up providing a git repo link than
> manually submitting patches to the mailing list.
Yeah, we've heard that proposal before. We're still not doing it though.
Insisting on patches being actually submitted to the mailing list is
important for archival and possibly legal reasons. If someone sends
in a link to $random-repo, once that site goes away there's no way to
determine exactly what was submitted.
regards, tom lane
On 3/11/17 2:06 PM, Tom Lane wrote: > Jim Nasby <jim.nasby@openscg.com> writes: >> It's actually a lot harder to mess up providing a git repo link than >> manually submitting patches to the mailing list. > Yeah, we've heard that proposal before. We're still not doing it though. > Insisting on patches being actually submitted to the mailing list is > important for archival and possibly legal reasons. If someone sends > in a link to $random-repo, once that site goes away there's no way to > determine exactly what was submitted. The full proposal was that the commitfest app have the ability to generate and post the patch for you, assuming that the smoke-test passes. -- Jim Nasby, Chief Data Architect, OpenSCG http://OpenSCG.com
On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote:
> FWIW, +1 on improving matters here.
>
> Andres Freund wrote:
>
> > The best I can come up so far is a toplevel target that creates the temp
> > install, starts a cluster and then runs the 'installcheck-or-check'
> > target on all the subdirectories via recursion. Individual makefiles can
> > either use the pre-existing cluster (most of of contrib for example), or
> > use the temporary install and run their pre-existing check target using
> > that (the tap tests, test_decoding, ...).
>
> I think a toplevel installcheck-or-check target is a good first step
> (though definitely lets find a better name). Just being able to run all
> tests without the need for 95% of pointless initdb's would be helpful
> enough.
Here's a hacky proof-of-concept patch that implements a contrib/
'fastcheck' target.
timing of:
make -s -j01 -Otarget check 2m49.286s
make -s -j16 -Otarget check 0m44.120s
make -s -j01 -Otarget fastcheck 1m1.533s
make -s -j16 -Otarget fastcheck 0m24.385s
make -s -j01 -Otarget USE_MODULE_DB=1 \
installcheck check-test_decoding-recurse 0m56.016s
make -s -j16 -Otarget USE_MODULE_DB=1 \
installcheck check-test_decoding-recurse 0m23.608s
All of these should, excepting rerunning initdb/server start, have
equivalent test coverage.
'fastcheck' is obviously, if you look at it, a POC. We'd need:
- abstract this away somehow, it's not nice to copy the logic into a
makefile
- tie contrib/'fastcheck' into check-world by overriding that target
- use the temp install we've created previously?
- more bulletproof server stop mechanism?
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
On 2017-03-11 14:17:42 -0800, Andres Freund wrote: > On 2017-03-07 09:36:51 -0300, Alvaro Herrera wrote: > > FWIW, +1 on improving matters here. > > > > Andres Freund wrote: > > > > > The best I can come up so far is a toplevel target that creates the temp > > > install, starts a cluster and then runs the 'installcheck-or-check' > > > target on all the subdirectories via recursion. Individual makefiles can > > > either use the pre-existing cluster (most of of contrib for example), or > > > use the temporary install and run their pre-existing check target using > > > that (the tap tests, test_decoding, ...). > > > > I think a toplevel installcheck-or-check target is a good first step > > (though definitely lets find a better name). Just being able to run all > > tests without the need for 95% of pointless initdb's would be helpful > > enough. > > Here's a hacky proof-of-concept patch that implements a contrib/ > 'fastcheck' target. > timing of: > > make -s -j01 -Otarget check 2m49.286s > make -s -j16 -Otarget check 0m44.120s > make -s -j01 -Otarget fastcheck 1m1.533s > make -s -j16 -Otarget fastcheck 0m24.385s > make -s -j01 -Otarget USE_MODULE_DB=1 \ > installcheck check-test_decoding-recurse 0m56.016s > make -s -j16 -Otarget USE_MODULE_DB=1 \ > installcheck check-test_decoding-recurse 0m23.608s Ooops - all these timings are from a coverage enabled build - the times are overall a bit smaller without that. - Andres
On 2017-03-11 11:48:31 -0800, Andres Freund wrote: > On 2017-03-11 12:05:23 -0500, Tom Lane wrote: > > I wrote: > > > I believe the core problem is that contrib/test_decoding's regresscheck > > > and isolationcheck targets each want to use ./tmp_check as their > > > --temp-instance. make has no reason to believe it shouldn't run those > > > two sub-jobs in parallel, but when it does, we get two postmasters trying > > > to share the same directory. This looks reasonably straightforward to > > > solve, but I'm not entirely familiar with the code here, and am not > > > sure what is the least ugly way to fix it. > > > > Enlarging on that: if I cd into contrib/test_decoding and do > > "make check -j4" or so, it reliably fails. > > Yep, can reproduce here as well. Interesting that, with -j16, I could > survive several dozen runs from the toplevel locally. > > > > This is a localized patch that only fixes things for > > contrib/test_decoding; what I'm wondering is if it would be better to > > establish a more widespread convention that > > $(pg_isolation_regress_check) should use a different --temp-instance > > directory than $(pg_regress_check) does. > > I think that'd be a good plan. We probably should also keep --outputdir > seperate (which test_decoding/Makefile does, but > pg_isolation_regress_check doesn't)? Here's a patch doing that (based on yours). I'd be kind of inclined to set --outputdir for !isolation tests too; possibly even move tmp_check below output_iso/ output_regress/ or such - but that seems like it potentially could cause some disagreement... - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
Andres Freund <andres@anarazel.de> writes:
> On 2017-03-11 11:48:31 -0800, Andres Freund wrote:
>> I think that'd be a good plan. We probably should also keep --outputdir
>> seperate (which test_decoding/Makefile does, but
>> pg_isolation_regress_check doesn't)?
> Here's a patch doing that (based on yours). I'd be kind of inclined to
> set --outputdir for !isolation tests too; possibly even move tmp_check
> below output_iso/ output_regress/ or such - but that seems like it
> potentially could cause some disagreement...
This looks generally sane to me, although I'm not very happy about folding
the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
seems weird and unlike the way it's done for the regular regression test
case. But probably Peter has a better-informed opinion.
regards, tom lane
0;115;0c On 2017-03-11 22:14:07 -0500, Tom Lane wrote: > Andres Freund <andres@anarazel.de> writes: > > On 2017-03-11 11:48:31 -0800, Andres Freund wrote: > >> I think that'd be a good plan. We probably should also keep --outputdir > >> seperate (which test_decoding/Makefile does, but > >> pg_isolation_regress_check doesn't)? > > > Here's a patch doing that (based on yours). I'd be kind of inclined to > > set --outputdir for !isolation tests too; possibly even move tmp_check > > below output_iso/ output_regress/ or such - but that seems like it > > potentially could cause some disagreement... > > This looks generally sane to me, although I'm not very happy about folding > the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that > seems weird and unlike the way it's done for the regular regression test > case. Yea, not super happy about that either - alternatively we could fold it into pg_regress. But given the way that prove_checks works, I thought it'd not be too ugly comparatively. - Andres
Andres Freund <andres@anarazel.de> writes:
> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>> This looks generally sane to me, although I'm not very happy about folding
>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>> seems weird and unlike the way it's done for the regular regression test
>> case.
> Yea, not super happy about that either - alternatively we could fold it
> into pg_regress.
Yeah, teaching pg_regress to auto-create the --temp-instance directory
seems perfectly sane from here.
regards, tom lane
On 03/13/2017 12:35 AM, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
>>> This looks generally sane to me, although I'm not very happy about folding
>>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
>>> seems weird and unlike the way it's done for the regular regression test
>>> case.
>> Yea, not super happy about that either - alternatively we could fold it
>> into pg_regress.
> Yeah, teaching pg_regress to auto-create the --temp-instance directory
> seems perfectly sane from here.
w.r.t. $subject, I thought it might be useful to get some recent stats
from the buildfarm. Results are below. The bin checks dwarf everything
else. Upgrade checks and isolation check are other items of significant
cost. Upgrade checks could be significantly shortened if we could avoid
rerunning the regression tests.
cheers
andrew
pgbfprod=> select s.branch, s.log_stage, count(*), avg(extract(epoch
from stage_duration)::numeric(15,2))::numeric(15,1),
stddev(extract(epoch from stage_duration)::numeric(15,2))::numeric(15,1)
from public.build_status_log s where sysname <> 'crake' and snapshot >
now() - interval '10 days' and log_stage !~ 'start|stop' group by
s.branch, s.log_stage having count(*) > 20 and avg(extract(epoch from
stage_duration)::numeric(15,2)) > 20 order by log_stage, branch <>
'HEAD', branch desc
; branch | log_stage | count
| avg | stddev
---------------+-----------------------------------------------+-------+--------+--------HEAD | bin-check.log
| 388 |
1739.0 | 1758.5REL9_6_STABLE | bin-check.log | 91 |
1430.5 | 1287.9REL9_5_STABLE | bin-check.log | 87 |
1140.0 | 994.1REL9_4_STABLE | bin-check.log | 72
| 751.0 | 666.8HEAD | check.log | 2305
| 263.1 | 1197.0REL9_6_STABLE | check.log | 610
| 294.7 | 1369.6REL9_5_STABLE | check.log | 627
| 170.1 | 819.6REL9_4_STABLE | check.log | 512
| 140.4 | 535.3REL9_3_STABLE | check.log | 449
| 112.0 | 446.0REL9_2_STABLE | check.log | 406
| 109.2 | 380.9HEAD | check-pg_upgrade.log | 1785
| 319.4 | 1310.5REL9_6_STABLE | check-pg_upgrade.log | 482
| 571.3 | 2811.0REL9_5_STABLE | check-pg_upgrade.log | 484
| 350.5 | 2160.3REL9_4_STABLE | check-pg_upgrade.log | 385
| 240.8 | 1278.9REL9_3_STABLE | check-pg_upgrade.log | 353
| 214.0 | 1188.3REL9_2_STABLE | check-pg_upgrade.log | 314
| 195.6 | 1016.6HEAD | config.log | 2216
| 84.5 | 101.5REL9_6_STABLE | config.log | 576
| 90.0 | 90.8REL9_5_STABLE | config.log | 584
| 114.0 | 358.8REL9_4_STABLE | config.log | 495
| 84.5 | 85.3REL9_3_STABLE | config.log | 431
| 97.9 | 100.7REL9_2_STABLE | config.log | 391
| 93.1 | 94.6HEAD | contrib-install-check-C.log | 2250
| 122.9 | 474.5REL9_6_STABLE | contrib-install-check-C.log | 606
| 124.6 | 410.3REL9_5_STABLE | contrib-install-check-C.log | 622
| 84.7 | 348.8REL9_4_STABLE | contrib-install-check-C.log | 508
| 105.9 | 434.1REL9_3_STABLE | contrib-install-check-C.log | 445
| 61.8 | 273.4REL9_2_STABLE | contrib-install-check-C.log | 403
| 54.3 | 205.7HEAD | contrib-install-check-cs_CZ.UTF-8.log | 184
| 25.7 | 11.8REL9_6_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 46
| 25.7 | 14.0REL9_5_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 42
| 23.8 | 16.6REL9_4_STABLE | contrib-install-check-cs_CZ.UTF-8.log | 36
| 22.9 | 12.4HEAD | contrib-install-check-en_US.8859-15.log | 37
| 173.9 | 32.3HEAD | contrib-install-check-en_US.ISO8859-1.log | 33
| 244.7 | 35.7HEAD | contrib-install-check-en_US.log | 171
| 65.9 | 101.6REL9_6_STABLE | contrib-install-check-en_US.log | 42
| 61.7 | 89.3REL9_5_STABLE | contrib-install-check-en_US.log | 37
| 54.3 | 79.3REL9_4_STABLE | contrib-install-check-en_US.log | 33
| 53.7 | 71.5REL9_3_STABLE | contrib-install-check-en_US.log | 32
| 57.8 | 86.6REL9_2_STABLE | contrib-install-check-en_US.log | 29
| 46.9 | 63.8HEAD | contrib-install-check-en_US.utf8.log | 142
| 29.2 | 31.4REL9_6_STABLE | contrib-install-check-en_US.utf8.log | 30
| 40.2 | 43.7REL9_5_STABLE | contrib-install-check-en_US.utf8.log | 29
| 41.8 | 51.6REL9_4_STABLE | contrib-install-check-en_US.utf8.log | 25
| 56.6 | 75.8REL9_3_STABLE | contrib-install-check-en_US.utf8.log | 24
| 35.7 | 40.4REL9_2_STABLE | contrib-install-check-en_US.utf8.log | 21
| 31.3 | 36.1HEAD | contrib-install-check-en_US.UTF-8.log | 188
| 79.7 | 73.8REL9_6_STABLE | contrib-install-check-en_US.UTF-8.log | 36
| 78.2 | 68.6REL9_5_STABLE | contrib-install-check-en_US.UTF-8.log | 34
| 64.9 | 53.7REL9_4_STABLE | contrib-install-check-en_US.UTF-8.log | 30
| 68.4 | 56.2REL9_3_STABLE | contrib-install-check-en_US.UTF-8.log | 28
| 54.6 | 46.1REL9_2_STABLE | contrib-install-check-en_US.UTF-8.log | 25
| 48.3 | 39.8HEAD | contrib-install-check-POSIX.log | 33
| 227.7 | 42.6HEAD | ecpg-check.log | 2229
| 88.5 | 212.4REL9_6_STABLE | ecpg-check.log | 586
| 97.5 | 139.6REL9_5_STABLE | ecpg-check.log | 591
| 92.5 | 183.5REL9_4_STABLE | ecpg-check.log | 491
| 82.1 | 117.1REL9_3_STABLE | ecpg-check.log | 428
| 77.3 | 102.8REL9_2_STABLE | ecpg-check.log | 388
| 80.7 | 98.3REL9_6_STABLE | initdb-C.log | 606
| 21.2 | 73.1HEAD | initdb-en_US.8859-15.log | 37
| 241.8 | 24.8HEAD | initdb-en_US.ISO8859-1.log | 33
| 251.6 | 39.1HEAD | initdb-en_US.log | 171
| 54.3 | 103.8REL9_6_STABLE | initdb-en_US.log | 42
| 54.9 | 99.7REL9_5_STABLE | initdb-en_US.log | 37
| 61.2 | 112.5REL9_4_STABLE | initdb-en_US.log | 33
| 47.5 | 78.5REL9_3_STABLE | initdb-en_US.log | 32
| 46.0 | 74.6REL9_2_STABLE | initdb-en_US.log | 29
| 41.1 | 62.9HEAD | initdb-POSIX.log | 33
| 251.1 | 31.6HEAD | install-check-C.log | 2241
| 229.5 | 1227.9REL9_6_STABLE | install-check-C.log | 606
| 299.0 | 1601.9REL9_5_STABLE | install-check-C.log | 622
| 173.7 | 1147.2REL9_4_STABLE | install-check-C.log | 508
| 139.3 | 796.4REL9_3_STABLE | install-check-C.log | 446
| 104.7 | 713.3REL9_2_STABLE | install-check-C.log | 403
| 91.3 | 574.6HEAD | install-check-cs_CZ.ISO-8859-2.log | 137
| 37.5 | 1.5REL9_6_STABLE | install-check-cs_CZ.ISO-8859-2.log | 33
| 30.8 | 1.5REL9_5_STABLE | install-check-cs_CZ.ISO-8859-2.log | 29
| 31.0 | 1.7REL9_4_STABLE | install-check-cs_CZ.ISO-8859-2.log | 25
| 26.2 | 0.4REL9_3_STABLE | install-check-cs_CZ.ISO-8859-2.log | 24
| 24.9 | 1.2REL9_2_STABLE | install-check-cs_CZ.ISO-8859-2.log | 21
| 22.6 | 0.7HEAD | install-check-cs_CZ.UTF-8.log | 184
| 43.0 | 9.7REL9_6_STABLE | install-check-cs_CZ.UTF-8.log | 46
| 39.1 | 15.1REL9_5_STABLE | install-check-cs_CZ.UTF-8.log | 42
| 39.4 | 14.6REL9_4_STABLE | install-check-cs_CZ.UTF-8.log | 36
| 32.3 | 9.5REL9_3_STABLE | install-check-cs_CZ.UTF-8.log | 34
| 29.7 | 8.2REL9_2_STABLE | install-check-cs_CZ.UTF-8.log | 30
| 27.6 | 9.2HEAD | install-check-cs_CZ.WIN-1250.log | 137
| 37.3 | 1.2REL9_6_STABLE | install-check-cs_CZ.WIN-1250.log | 33
| 30.5 | 1.3REL9_5_STABLE | install-check-cs_CZ.WIN-1250.log | 29
| 31.0 | 1.7REL9_4_STABLE | install-check-cs_CZ.WIN-1250.log | 25
| 26.3 | 0.7REL9_3_STABLE | install-check-cs_CZ.WIN-1250.log | 24
| 24.9 | 1.4REL9_2_STABLE | install-check-cs_CZ.WIN-1250.log | 21
| 22.3 | 0.6HEAD | install-check-de_DE.utf8.log | 129
| 35.4 | 9.4REL9_6_STABLE | install-check-de_DE.utf8.log | 22
| 25.8 | 7.8REL9_5_STABLE | install-check-de_DE.utf8.log | 21
| 27.4 | 8.0HEAD | install-check-en_US.8859-15.log | 37
| 136.9 | 21.2HEAD | install-check-en_US.ISO8859-1.log | 33
| 319.0 | 29.4HEAD | install-check-en_US.log | 171
| 65.7 | 58.7REL9_6_STABLE | install-check-en_US.log | 42
| 62.7 | 69.4REL9_5_STABLE | install-check-en_US.log | 37
| 59.4 | 56.2REL9_4_STABLE | install-check-en_US.log | 33
| 50.6 | 44.7REL9_3_STABLE | install-check-en_US.log | 32
| 50.2 | 46.8REL9_2_STABLE | install-check-en_US.log | 29
| 45.1 | 39.2HEAD | install-check-en_US.utf8.log | 142
| 47.6 | 40.3REL9_6_STABLE | install-check-en_US.utf8.log | 30
| 58.1 | 54.4REL9_5_STABLE | install-check-en_US.utf8.log | 29
| 61.7 | 57.4REL9_4_STABLE | install-check-en_US.utf8.log | 25
| 57.2 | 59.9REL9_3_STABLE | install-check-en_US.utf8.log | 24
| 51.3 | 47.7REL9_2_STABLE | install-check-en_US.utf8.log | 21
| 45.2 | 41.9HEAD | install-check-en_US.UTF-8.log | 188
| 153.5 | 162.7REL9_6_STABLE | install-check-en_US.UTF-8.log | 36
| 140.8 | 140.9REL9_5_STABLE | install-check-en_US.UTF-8.log | 34
| 145.7 | 144.3REL9_4_STABLE | install-check-en_US.UTF-8.log | 30
| 118.9 | 113.1REL9_3_STABLE | install-check-en_US.UTF-8.log | 28
| 107.0 | 101.8REL9_2_STABLE | install-check-en_US.UTF-8.log | 25
| 93.6 | 87.8HEAD | install-check-POSIX.log | 33
| 189.5 | 76.5HEAD | install-check-sk_SK.ISO-8859-2.log | 137
| 37.4 | 1.2REL9_6_STABLE | install-check-sk_SK.ISO-8859-2.log | 33
| 30.5 | 1.3REL9_5_STABLE | install-check-sk_SK.ISO-8859-2.log | 29
| 30.9 | 1.5REL9_4_STABLE | install-check-sk_SK.ISO-8859-2.log | 25
| 26.2 | 0.4REL9_3_STABLE | install-check-sk_SK.ISO-8859-2.log | 24
| 24.8 | 1.2REL9_2_STABLE | install-check-sk_SK.ISO-8859-2.log | 21
| 22.7 | 0.6HEAD | install-check-sk_SK.UTF-8.log | 137
| 37.6 | 1.1REL9_6_STABLE | install-check-sk_SK.UTF-8.log | 33
| 31.0 | 1.8REL9_5_STABLE | install-check-sk_SK.UTF-8.log | 29
| 31.3 | 1.8REL9_4_STABLE | install-check-sk_SK.UTF-8.log | 25
| 26.2 | 0.4REL9_3_STABLE | install-check-sk_SK.UTF-8.log | 24
| 25.0 | 1.2REL9_2_STABLE | install-check-sk_SK.UTF-8.log | 21
| 22.7 | 1.2HEAD | install-check-sk_SK.WIN-1250.log | 137
| 37.4 | 1.2REL9_6_STABLE | install-check-sk_SK.WIN-1250.log | 33
| 30.4 | 0.7REL9_5_STABLE | install-check-sk_SK.WIN-1250.log | 29
| 31.0 | 1.7REL9_4_STABLE | install-check-sk_SK.WIN-1250.log | 25
| 26.2 | 0.4REL9_3_STABLE | install-check-sk_SK.WIN-1250.log | 24
| 25.0 | 1.2REL9_2_STABLE | install-check-sk_SK.WIN-1250.log | 21
| 22.8 | 1.1HEAD | isolation-check.log | 2250
| 175.6 | 457.5REL9_6_STABLE | isolation-check.log | 606
| 201.0 | 527.7REL9_5_STABLE | isolation-check.log | 622
| 172.3 | 846.7REL9_4_STABLE | isolation-check.log | 508
| 146.0 | 602.3REL9_3_STABLE | isolation-check.log | 445
| 114.7 | 504.4REL9_2_STABLE | isolation-check.log | 403
| 57.2 | 300.6HEAD | make-contrib.log | 2176
| 43.7 | 67.6REL9_6_STABLE | make-contrib.log | 572
| 46.2 | 50.3REL9_5_STABLE | make-contrib.log | 579
| 44.9 | 54.2REL9_4_STABLE | make-contrib.log | 491
| 50.0 | 52.3REL9_3_STABLE | make-contrib.log | 428
| 48.6 | 51.4REL9_2_STABLE | make-contrib.log | 388
| 42.2 | 42.8HEAD | make-dist.log | 36
| 56.7 | 0.7HEAD | make-doc.log | 36
| 150.8 | 1.5REL9_2_STABLE | make-install.log | 403
| 22.0 | 29.8HEAD | make.log | 2328
| 363.1 | 581.0REL9_6_STABLE | make.log | 610
| 369.6 | 400.9REL9_5_STABLE | make.log | 627
| 392.8 | 556.8REL9_4_STABLE | make.log | 512
| 335.3 | 329.4REL9_3_STABLE | make.log | 449
| 320.0 | 311.0REL9_2_STABLE | make.log | 406
| 299.8 | 276.3HEAD | pl-install-check-en_US.8859-15.log | 37
| 32.9 | 3.3HEAD | pl-install-check-en_US.ISO8859-1.log | 33
| 34.3 | 5.1HEAD | pl-install-check-POSIX.log | 33
| 33.1 | 8.3REL9_2_STABLE | SCM-checkout.log | 406
| 33.2 | 87.8HEAD | sepgsql-test.log | 63
| 20.7 | 6.8HEAD | test-decoding-check.log | 1905
| 144.9 | 770.6REL9_6_STABLE | test-decoding-check.log | 493
| 226.3 | 1244.0REL9_5_STABLE | test-decoding-check.log | 495
| 147.1 | 1080.2REL9_4_STABLE | test-decoding-check.log | 419
| 168.1 | 1117.6HEAD | testmodules-install-check-C.log | 2233
| 22.6 | 42.3REL9_6_STABLE | testmodules-install-check-C.log | 602
| 25.6 | 35.8HEAD | testmodules-install-check-en_US.8859-15.log | 37
| 57.8 | 5.2HEAD | testmodules-install-check-en_US.ISO8859-1.log | 33
| 58.9 | 10.6HEAD | testmodules-install-check-en_US.log | 171
| 20.2 | 28.7REL9_6_STABLE | testmodules-install-check-en_US.utf8.log | 30
| 30.4 | 46.9HEAD | testmodules-install-check-en_US.UTF-8.log | 188
| 22.8 | 19.4REL9_6_STABLE | testmodules-install-check-en_US.UTF-8.log | 36
| 23.6 | 19.4REL9_5_STABLE | testmodules-install-check-en_US.UTF-8.log | 34
| 20.4 | 16.4HEAD | testmodules-install-check-POSIX.log | 33
| 72.1 | 14.7HEAD | typedefs.log | 96
| 74.8 | 99.1REL9_6_STABLE | typedefs.log | 28
| 106.3 | 119.7REL9_5_STABLE | typedefs.log | 28
| 100.0 | 111.7REL9_4_STABLE | typedefs.log | 24
| 88.1 | 100.7REL9_3_STABLE | typedefs.log | 23
| 85.6 | 95.1
(180 rows)
--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 03/13/2017 07:21 PM, Andrew Dunstan wrote: > > On 03/13/2017 12:35 AM, Tom Lane wrote: >> Andres Freund <andres@anarazel.de> writes: >>> On 2017-03-11 22:14:07 -0500, Tom Lane wrote: >>>> This looks generally sane to me, although I'm not very happy about folding >>>> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that >>>> seems weird and unlike the way it's done for the regular regression test >>>> case. >>> Yea, not super happy about that either - alternatively we could fold it >>> into pg_regress. >> Yeah, teaching pg_regress to auto-create the --temp-instance directory >> seems perfectly sane from here. > > w.r.t. $subject, I thought it might be useful to get some recent stats > from the buildfarm. Results are below. The bin checks dwarf everything > else. Upgrade checks and isolation check are other items of significant > cost. Upgrade checks could be significantly shortened if we could avoid > rerunning the regression tests. Sorry for line wrapping. Better formatted here: <https://gist.github.com/adunstan/fbbdfe646427710233ecb67569665de9> cheers andrew -- Andrew Dunstan https://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
On 2017-03-13 00:35:06 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2017-03-11 22:14:07 -0500, Tom Lane wrote:
> >> This looks generally sane to me, although I'm not very happy about folding
> >> the "$(MKDIR_P) output_iso" call into pg_isolation_regress_check --- that
> >> seems weird and unlike the way it's done for the regular regression test
> >> case.
>
> > Yea, not super happy about that either - alternatively we could fold it
> > into pg_regress.
>
> Yeah, teaching pg_regress to auto-create the --temp-instance directory
> seems perfectly sane from here.
I was thinking about outputdir, not temp-instance. The latter is
already created:
/* make the temp instance top directory */
make_directory(temp_instance);
Attached is an updated patch that creates outputdir if necessary. This
is possibly going to trigger a time-to-check-time-to-use coverity
warning, but the rest of pg_regress does if(!exists) mkdir() type logic,
so I did the same.
Besides the pg_regress change, the only thing I've changed is to remove
the in-line "$(MKDIR_P) output_iso && \".
- Andres
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Attachment
make check-world -O -j6 PROVE_FLAGS=-j6 2 min 34 seconds Nice! -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Peter Eisentraut <peter.eisentraut@2ndquadrant.com> writes:
> make check-world -O -j6 PROVE_FLAGS=-j6
> 2 min 34 seconds
> Nice!
One thing I've noticed with parallel check-world is that it's possible
for a sub-job to fail and for the resulting complaint from make to scroll
offscreen before everything stops, leaving all the visible output being
from another sub-job that completed just fine. If you don't look VERY
closely, or check $?, you can easily be misled into thinking the tests
all passed.
I think it'd be a good idea to fix check-world so that there's a positive
printout at the end of a successful run, along the lines of the
+@echo "All of PostgreSQL successfully made. Ready to install."
we have for the "all" target.
regards, tom lane