Thread: More efficient build farm animal wakeup?
Hi, Is there a way to find out about new git commits that is more efficient and timely than running N git fetches or whatever every minute in a cron job? Maybe some kind of long polling where you send an HTTP request that says "I think the tips of branches x, y, z are at 111, 222, 333" and the server responds when that ceases to be true?
On Sat, Nov 19, 2022 at 4:13 AM Thomas Munro <thomas.munro@gmail.com> wrote:
Hi,
Is there a way to find out about new git commits that is more
efficient and timely than running N git fetches or whatever every
minute in a cron job? Maybe some kind of long polling where you send
an HTTP request that says "I think the tips of branches x, y, z are at
111, 222, 333" and the server responds when that ceases to be true?
I'm not aware of any such thing standardized for git, but it wouldn't be hard to build one for that (I'm talking primarily about the server side here, not how to integrate that into the buildfarm side of things).
We could also set something up whereby we could fire off webhooks when branches change (easy enough for registered servers in the buildfarm as we can easily avoid abuse there -- it would take more work to make something like that a public service, due to the risk of abuse). But that may actually be worse off, since I bet a lot of buildfarm animals (most even?) are probably sitting behind a NAT gateway of some kind, meaning consuming webhooks is hard.
I did something similar for how we did things on borka (using some internal pginfra webhooks that are not available to the public at this time), but I had to revert that because of issues with concurrent buildfarm runs in the environment that we had set up. But we are using it for the snapshots docs builder, to make sure the website for that gets updated immediately after a commit on master. But the principle definitely work.
Another thing to consider would be that something like this would cause all buildfarm clients to start git pull:ing down changes at more or less exactly the same time. Though in total that would probably still mean a lot less load than those that "git pul" very frequently today, it could potentially lead to some nets with lots of bf clients experiencing some level of bandwidth filling or something. Could probably be solved pretty easily with a random delay (which doesn't have to be long, as for most git pulls it will be a very quick operation), just something that's worth considering.
tl,tr; it's not there now, but yes if we can find a smart way for th ebf clients to consume it, it is something we could build and deploy fairly easily.
On 2022-11-18 Fr 22:12, Thomas Munro wrote: > Hi, > > Is there a way to find out about new git commits that is more > efficient and timely than running N git fetches or whatever every > minute in a cron job? Maybe some kind of long polling where you send > an HTTP request that says "I think the tips of branches x, y, z are at > 111, 222, 333" and the server responds when that ceases to be true? It might not suit your use case, but one of the things I do to reduce fetch load is to run a local mirror which runs git fetch -q --prune every 5 minutes. It also runs a git daemon, and several of my animals point at that. If there's a better git API I'll be happy to try to use it. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Sun, Nov 20, 2022 at 1:35 AM Magnus Hagander <magnus@hagander.net> wrote: > tl,tr; it's not there now, but yes if we can find a smart way for th ebf clients to consume it, it is something we couldbuild and deploy fairly easily. Cool -- it sounds a lot like you've thought about this already :-) About the client: currently run_branches.pl makes an HTTP request for the "branches of interest" list. Seems like a candidate point for a long poll? I don't think it'd have to be much smarter than it is today, it'd just have to POST the commits it already has, I think. Perhaps as a first step, the server could immediately report which branches to bother fetching, considering the client's existing commits. That'd almost always be none, but ~11.7 times per day a new commit shows up, and once a year there's a new interesting branch. That would avoid the need for the 6 git fetches that usually follow in the common case, which admittedly might not be a change worth making on its own. After all, the git fetches are probably quite similar HTTP requests themselves, except that there 6 of them, one per branch, and they hit the public git server instead of some hypothetical buildfarm endpoint. Then you could switch to long polling by letting the client say "if currently none, I'm prepared to wait up to X seconds for a different answer", assuming you know how to build the server side of that (insert magic here). Of course, you can't make it too long or your session might be dropped in the badlands between client and server, but that's just a reason to make X configurable. I think RFC6202 says that 120 seconds probably works fine across most kinds of links, which means that you lower the total poll rate hitting the server, but--more interestingly for me as a client--you minimise latency when something finally happens. (With various keepalive tricks and/or heartbeat streaming tricks you could possibly make it much higher, who knows... but you'd have to set it very very low to do worse than what we're doing today in total request count). Or maybe there is some existing easy perl library that could be used for this (joke answer: cpan install Twitter::API and follow @pg_commits). By the way, the reason I wrote this is because I've just been re-establishing my animal elver. It's set to run every minute by cron, and spends nearly *half of each minute* running various git commands when nothing is happening. Actually it's more than 6 connections to the server, because I see there's a fetch and an ls-remote, so it's at least 12 (being unfamiliar with git plumbing, it could be much more for all I know, and I kinda suspect so based on the total run time). Admittedly network packets take a little while to fly to my South Pacific location so maybe this looks less insane from over there. However, when I started this thread I was half expecting such a thing to exist already, somewhere, I just haven't been able to find it myself... Don't other people have this problem? Maybe everybody who has this problem uses webhooks (git server post commit hook opens connection to client) as you mentioned, but as you also mentioned that'd never fly for our topology.
Hi, On 2022-11-19 16:12:24 +1300, Thomas Munro wrote: > Is there a way to find out about new git commits that is more > efficient and timely than running N git fetches or whatever every > minute in a cron job? Maybe some kind of long polling where you send > an HTTP request that says "I think the tips of branches x, y, z are at > 111, 222, 333" and the server responds when that ceases to be true? I think a git fetch is actually ok for that - it doesn't take a whole lot of resources. However run_builds.pl is more heavyweight. For one, it starts one run_build.pl for each branch, which each then fetches from git separately. But more importantly, each run_build.pl seems to actually do a fair bit of work before discovering nothing has changed. A typical log I see: Nov 20 06:08:17 bf-valgrind-v4 run_branches.pl[3289916]: Sun Nov 20 06:08:17 2022: buildfarm run for grassquit:REL_14_STABLEstarting Nov 20 06:08:17 bf-valgrind-v4 run_branches.pl[3289916]: grassquit:REL_14_STABLE [06:08:17] checking out source ... Nov 20 06:08:20 bf-valgrind-v4 run_branches.pl[3289916]: grassquit:REL_14_STABLE [06:08:20] checking if build run needed... Nov 20 06:08:20 bf-valgrind-v4 run_branches.pl[3289916]: grassquit:REL_14_STABLE [06:08:20] No build required: last status= Sat Nov 19 23:54:38 2022 GMT, cur> So we spend three seconds in the "checking out source" stage, just to then see that nothing has actually changed. Greetings, Andres Freund
On Sun, Nov 20, 2022 at 4:56 AM Thomas Munro <thomas.munro@gmail.com> wrote:
On Sun, Nov 20, 2022 at 1:35 AM Magnus Hagander <magnus@hagander.net> wrote:
> tl,tr; it's not there now, but yes if we can find a smart way for th ebf clients to consume it, it is something we could build and deploy fairly easily.
Cool -- it sounds a lot like you've thought about this already :-)
About the client: currently run_branches.pl makes an HTTP request for
the "branches of interest" list. Seems like a candidate point for a
long poll? I don't think it'd have to be much smarter than it is
today, it'd just have to POST the commits it already has, I think.
Um, branches of interest will only pick up when it gets a new *branch*, not a new *commit*, so I think that would be a very different problem to solve. And I don't think we have new branche *that* often...
Perhaps as a first step, the server could immediately report which
branches to bother fetching, considering the client's existing
commits. That'd almost always be none, but ~11.7 times per day a new
commit shows up, and once a year there's a new interesting branch.
That would avoid the need for the 6 git fetches that usually follow in
the common case, which admittedly might not be a change worth making
on its own. After all, the git fetches are probably quite similar
HTTP requests themselves, except that there 6 of them, one per branch,
and they hit the public git server instead of some hypothetical
buildfarm endpoint.
As Andres mentioned downthread, that's not a lot more lightweight than what "git fetch" does.
The thing we'd want to avoid is having to do that so much and often. And getting to that is going to require modification of the buildfarm client to make it more "smart" regardless. In particular, making it do this "right" in the face of multiple branches is probably going to be a big win.
Then you could switch to long polling by letting the client say "if
currently none, I'm prepared to wait up to X seconds for a different
answer", assuming you know how to build the server side of that
(insert magic here). Of course, you can't make it too long or your
session might be dropped in the badlands between client and server,
but that's just a reason to make X configurable. I think RFC6202 says
that 120 seconds probably works fine across most kinds of links, which
means that you lower the total poll rate hitting the server, but--more
interestingly for me as a client--you minimise latency when something
finally happens. (With various keepalive tricks and/or heartbeat
streaming tricks you could possibly make it much higher, who knows...
but you'd have to set it very very low to do worse than what we're
doing today in total request count). Or maybe there is some existing
easy perl library that could be used for this (joke answer: cpan
install Twitter::API and follow @pg_commits).
I also honestly wonder how big a problem a much longer than 120 seconds timeout would be in practice. Since we own both the client and the server in this case, we'd only be at mercy of network equipment in between and I think we're much less exposed to weirdness there than "the average browser". Thus, as long as it's configurable, I think we could go for something much longer by default.
I'd imagine something like a
X-branch-master: a4adc31f69
X-branch-REL_14_STABLE: b33283cbd3
X-longpoll: 120
For that one it would check branch master and rel 14, and if either branchtip doesn't match what was in the header, it'd return immediately with a textfile that's basically
master:<whateveritis>
if master has changed and not REL_14.
If nothing has changed, go into longpoll for 120 seconds based on the header, and if nothing at all has changed in that time, return a 304.
We could also use something like a websocket to just stream the changes out over.
In either case it would also need to change the buildfarm client to run as a daemon rather than a cronjob I think? (obviously optional, we don't have to remove the current abilities)
However, when I started this thread I was half expecting such a thing
to exist already, somewhere, I just haven't been able to find it
myself... Don't other people have this problem? Maybe everybody who
has this problem uses webhooks (git server post commit hook opens
connection to client) as you mentioned, but as you also mentioned
that'd never fly for our topology.
Yeah, webhook seems to be what most people use.
FWIW, an implementation for us would be a small daemon that receives such webhooks from our git server and redistributtes it for the long polling. That's still the easiest way to get the data out of git itself...
//Magnus
On Mon, Nov 21, 2022 at 10:31 AM Magnus Hagander <magnus@hagander.net> wrote: > Um, branches of interest will only pick up when it gets a new *branch*, not a new *commit*, so I think that would be avery different problem to solve. And I don't think we have new branche *that* often... Sure, could be done with an extra different request you make from time to time or keeping the existing list. No strong opinions on that, I was just observing that it could also be combined, something like: Client: I have 14@1234, 15@1234, HEAD@1234; what should I do now, boss? Server: You should fetch 14 (it has a new commit) and 16 (it's a new branch you didn't mention). > I'd imagine something like a > GET https://git.postgresql.org/buildfarm-branchtips > X-branch-master: a4adc31f69 > X-branch-REL_14_STABLE: b33283cbd3 > X-longpoll: 120 > > For that one it would check branch master and rel 14, and if either branchtip doesn't match what was in the header, it'dreturn immediately with a textfile that's basically > master:<whateveritis> > > if master has changed and not REL_14. > > If nothing has changed, go into longpoll for 120 seconds based on the header, and if nothing at all has changed in thattime, return a 304. LGTM, that's exactly the sort of thing I was imagining. > We could also use something like a websocket to just stream the changes out over. True. The reason I started on about long polling instead of websockets is that I was imagining that the simpler, dumber protocol where the client doesn't even really know it's participating a new kind of magic would be more cromulent in ye olde perl script (no new cpan dependencies). > In either case it would also need to change the buildfarm client to run as a daemon rather than a cronjob I think? (obviouslyoptional, we don't have to remove the current abilities) Given that the point of the build farm is (these days) to test on weird computers and operating systems, I expect that proper 'run like a service' support would be painful or not get done. It'd be nice if there were some way to make this work with simple crontab entries...
On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan <andrew@dunslane.net> wrote: > It might not suit your use case, but one of the things I do to reduce > fetch load is to run a local mirror which runs > > git fetch -q --prune > > every 5 minutes. It also runs a git daemon, and several of my animals > point at that. Thanks. I understand now that my configuration without a local mirror is super inefficient (it spends the first ~25s of each minute running git commands). Still, even though that can be improved by me setting up more stuff, I'd like something event-driven rather than short polling-based for lower latency. > If there's a better git API I'll be happy to try to use it. Cool. Seems like we just have to invent something first... FWIW I'm also trying to chase the short polling out of cfbot. It regularly harasses the git servers at one end (could be fixed with this approach), and wastes a percentage of our allotted CPU slots on the other end by scheduling periodically (could be fixed with webhooks from Cirrus).
On 2022-11-20 Su 17:32, Thomas Munro wrote:
> On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>> It might not suit your use case, but one of the things I do to reduce
>> fetch load is to run a local mirror which runs
>>
>> git fetch -q --prune
>>
>> every 5 minutes. It also runs a git daemon, and several of my animals
>> point at that.
> Thanks. I understand now that my configuration without a local mirror
> is super inefficient (it spends the first ~25s of each minute running
> git commands). Still, even though that can be improved by me setting
> up more stuff, I'd like something event-driven rather than short
> polling-based for lower latency.
>
>> If there's a better git API I'll be happy to try to use it.
> Cool. Seems like we just have to invent something first...
>
> FWIW I'm also trying to chase the short polling out of cfbot. It
> regularly harasses the git servers at one end (could be fixed with
> this approach), and wastes a percentage of our allotted CPU slots on
> the other end by scheduling periodically (could be fixed with webhooks
> from Cirrus).
I think I have solved most of the actual issues without getting too complex.
Here's how:
The buildfarm server now creates a companion to branches_of_interest.txt
called branches_of_interest.json which looks like this:
[
{
"REL_11_STABLE" : "140c803723"
},
{
"REL_12_STABLE" : "4cbcb7ed85"
},
{
"REL_13_STABLE" : "c13667b518"
},
{
"REL_14_STABLE" : "5cda142bb9"
},
{
"REL_15_STABLE" : "ff9d27ee2b"
},
{
"HEAD" : "51b5834cd5"
}
]
It updates this every time it does a git fetch, currently every 5 minutes.
run_branches.pl fetches this file instead of the plain list of branches,
and before running run_build.pl checks if the given commit was the
latest one tested, and if so and a build isn't being forced, skips the
branch. Thus, in the case where all the branches are up to date there
will be no git calls whatsoever.
You can try it out by getting run_branches.pl from
<https://raw.githubusercontent.com/PGBuildFarm/client-code/main/run_branches.pl>
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
> The buildfarm server now creates a companion to branches_of_interest.txt
> called branches_of_interest.json which looks like this:
... okay ...
> It updates this every time it does a git fetch, currently every 5 minutes.
That up-to-five-minute delay, on top of whatever cronjob delay one has
on one's animals, seems kind of sad. I've gotten kind of spoiled maybe
by seeing first buildfarm results typically within 15 minutes of a push.
But if we're trying to improve matters in this area, this doesn't seem
like quite the way to go.
But it does seem like this eliminates one expense. Now that you have
that bit, maybe we could arrange a webhook or something that allows
branches_of_interest.json to get updated immediately after a push?
regards, tom lane
n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew@dunslane.net> writes:
> The buildfarm server now creates a companion to branches_of_interest.txt
> called branches_of_interest.json which looks like this:
... okay ...
Yeah, it's not as efficient as something like long polling or web sockets, but it is most definitely a lot simpler!
If we're going to have a lot of animals do pulls of this file every minute or more, it's certainly a lot better to pull this small file than to make multiple git calls.
It could trivially be made even more efficient by making the request with either a If-None-Match or If-Modified-Since. While it's still small, that cuts the size approximately in half, and would allow you to skip even more processing if nothing has changed.
> It updates this every time it does a git fetch, currently every 5 minutes.
That up-to-five-minute delay, on top of whatever cronjob delay one has
on one's animals, seems kind of sad. I've gotten kind of spoiled maybe
by seeing first buildfarm results typically within 15 minutes of a push.
But if we're trying to improve matters in this area, this doesn't seem
like quite the way to go.
But it does seem like this eliminates one expense. Now that you have
that bit, maybe we could arrange a webhook or something that allows
branches_of_interest.json to get updated immediately after a push?
Webhooks are definitely a lot easier to implement in between our servers yeah, so that shouldn't be too hard. We could use the same hooks that we use for borka to build the docs, but have it just run whatever script it is the buildfarm needs. I assume it's just something trivial to run there, Andrew?
//Magnus
On Mon, Nov 21, 2022 at 9:51 PM Andrew Dunstan <andrew@dunslane.net> wrote:
On 2022-11-20 Su 17:32, Thomas Munro wrote:
> On Sun, Nov 20, 2022 at 2:44 AM Andrew Dunstan <andrew@dunslane.net> wrote:
>> It might not suit your use case, but one of the things I do to reduce
>> fetch load is to run a local mirror which runs
>>
>> git fetch -q --prune
>>
>> every 5 minutes. It also runs a git daemon, and several of my animals
>> point at that.
> Thanks. I understand now that my configuration without a local mirror
> is super inefficient (it spends the first ~25s of each minute running
> git commands). Still, even though that can be improved by me setting
> up more stuff, I'd like something event-driven rather than short
> polling-based for lower latency.
>
>> If there's a better git API I'll be happy to try to use it.
> Cool. Seems like we just have to invent something first...
>
> FWIW I'm also trying to chase the short polling out of cfbot. It
> regularly harasses the git servers at one end (could be fixed with
> this approach), and wastes a percentage of our allotted CPU slots on
> the other end by scheduling periodically (could be fixed with webhooks
> from Cirrus).
I think I have solved most of the actual issues without getting too complex.
Here's how:
The buildfarm server now creates a companion to branches_of_interest.txt
called branches_of_interest.json which looks like this:
[
{
"REL_11_STABLE" : "140c803723"
},
{
"REL_12_STABLE" : "4cbcb7ed85"
},
{
"REL_13_STABLE" : "c13667b518"
},
{
"REL_14_STABLE" : "5cda142bb9"
},
{
"REL_15_STABLE" : "ff9d27ee2b"
},
{
"HEAD" : "51b5834cd5"
}
]
Is there a reason this file is a list of hashes each hash with a single value in it? Would it make more sense if it was:
{
"REL_11_STABLE": "140c803723",
"REL_12_STABLE": "4cbcb7ed85",
"REL_13_STABLE": "c13667b518",
"REL_14_STABLE": "5cda142bb9",
"REL_15_STABLE": "ff9d27ee2b",
"HEAD": "51b5834cd5"
}
"REL_11_STABLE": "140c803723",
"REL_12_STABLE": "4cbcb7ed85",
"REL_13_STABLE": "c13667b518",
"REL_14_STABLE": "5cda142bb9",
"REL_15_STABLE": "ff9d27ee2b",
"HEAD": "51b5834cd5"
}
On 2022-11-21 Mo 16:26, Magnus Hagander wrote:
>
> Is there a reason this file is a list of hashes each hash with a
> single value in it? Would it make more sense if it was:
> {
> "REL_11_STABLE": "140c803723",
> "REL_12_STABLE": "4cbcb7ed85",
> "REL_13_STABLE": "c13667b518",
> "REL_14_STABLE": "5cda142bb9",
> "REL_15_STABLE": "ff9d27ee2b",
> "HEAD": "51b5834cd5"
> }
>
No. It's the way it is because the client relies on their being in the
right order. JSON hashes are conceptually unordered.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
On Mon, Nov 21, 2022 at 11:27 PM Andrew Dunstan <andrew@dunslane.net> wrote:
On 2022-11-21 Mo 16:26, Magnus Hagander wrote:
>
> Is there a reason this file is a list of hashes each hash with a
> single value in it? Would it make more sense if it was:
> {
> "REL_11_STABLE": "140c803723",
> "REL_12_STABLE": "4cbcb7ed85",
> "REL_13_STABLE": "c13667b518",
> "REL_14_STABLE": "5cda142bb9",
> "REL_15_STABLE": "ff9d27ee2b",
> "HEAD": "51b5834cd5"
> }
>
No. It's the way it is because the client relies on their being in the
right order. JSON hashes are conceptually unordered.
Ah yeah, if they need to be ordered that certainly makes more sense.
On 2022-11-21 Mo 15:58, Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: >> The buildfarm server now creates a companion to branches_of_interest.txt >> called branches_of_interest.json which looks like this: > ... okay ... > >> It updates this every time it does a git fetch, currently every 5 minutes. > That up-to-five-minute delay, on top of whatever cronjob delay one has > on one's animals, seems kind of sad. I've gotten kind of spoiled maybe > by seeing first buildfarm results typically within 15 minutes of a push. > But if we're trying to improve matters in this area, this doesn't seem > like quite the way to go. Well, 5 minutes was originally chosen because it was sufficient for the purpose for which up to now the server used its mirror. Now we have added a new purpose we can certainly revisit that. Shall I try 2 minutes or go down to 1? > > But it does seem like this eliminates one expense. Now that you have > that bit, maybe we could arrange a webhook or something that allows > branches_of_interest.json to get updated immediately after a push? > > Sure, if you think and extra few seconds is worth saving. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
> On 2022-11-21 Mo 15:58, Tom Lane wrote:
>> But if we're trying to improve matters in this area, this doesn't seem
>> like quite the way to go.
> Well, 5 minutes was originally chosen because it was sufficient for the
> purpose for which up to now the server used its mirror. Now we have
> added a new purpose we can certainly revisit that. Shall I try 2 minutes
> or go down to 1?
Actually, if we implement a webhook to update this, the server could
stop doing speculative git pulls too, no?
regards, tom lane
On 2022-11-21 Mo 16:20, Magnus Hagander wrote: > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Andrew Dunstan <andrew@dunslane.net> writes: > > The buildfarm server now creates a companion to > branches_of_interest.txt > > called branches_of_interest.json which looks like this: > > ... okay ... > > > Yeah, it's not as efficient as something like long polling or web > sockets, but it is most definitely a lot simpler! > > If we're going to have a lot of animals do pulls of this file every > minute or more, it's certainly a lot better to pull this small file > than to make multiple git calls. > > It could trivially be made even more efficient by making the request > with either a If-None-Match or If-Modified-Since. While it's still > small, that cuts the size approximately in half, and would allow you > to skip even more processing if nothing has changed. I'll look at that. > > > > It updates this every time it does a git fetch, currently every > 5 minutes. > > That up-to-five-minute delay, on top of whatever cronjob delay one has > on one's animals, seems kind of sad. I've gotten kind of spoiled > maybe > by seeing first buildfarm results typically within 15 minutes of a > push. > But if we're trying to improve matters in this area, this doesn't seem > like quite the way to go. > > But it does seem like this eliminates one expense. Now that you have > that bit, maybe we could arrange a webhook or something that allows > branches_of_interest.json to get updated immediately after a push? > > > Webhooks are definitely a lot easier to implement in between our > servers yeah, so that shouldn't be too hard. We could use the same > hooks that we use for borka to build the docs, but have it just run > whatever script it is the buildfarm needs. I assume it's just > something trivial to run there, Andrew? Yes, I think much better between servers. Currently the cron job looks something like this: */5 * * * * cd $HOME/postgresql.git && git fetch -q && $HOME/website/bin/branches_of_interest.pl That script is what sets up the json files. I know nothing about git webhooks though, someone will have to point me in the right direction. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
On Mon, Nov 21, 2022 at 11:41 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew@dunslane.net> writes:
> On 2022-11-21 Mo 15:58, Tom Lane wrote:
>> But if we're trying to improve matters in this area, this doesn't seem
>> like quite the way to go.
> Well, 5 minutes was originally chosen because it was sufficient for the
> purpose for which up to now the server used its mirror. Now we have
> added a new purpose we can certainly revisit that. Shall I try 2 minutes
> or go down to 1?
Actually, if we implement a webhook to update this, the server could
stop doing speculative git pulls too, no?
That would be the main point, yes. Saves a few hundred (or thousand) wasteful git pulls *and* reacts quicker to actual pushes. As long as you have a clear line of communications between the machines, it's basically win/win I think. That's probably why, as Thomas noticed earlier, that's what "everybody" does.
On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan <andrew@dunslane.net> wrote:
On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
> n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andrew Dunstan <andrew@dunslane.net> writes:
> > The buildfarm server now creates a companion to
> branches_of_interest.txt
> > called branches_of_interest.json which looks like this:
>
> ... okay ...
>
>
> Yeah, it's not as efficient as something like long polling or web
> sockets, but it is most definitely a lot simpler!
>
> If we're going to have a lot of animals do pulls of this file every
> minute or more, it's certainly a lot better to pull this small file
> than to make multiple git calls.
>
> It could trivially be made even more efficient by making the request
> with either a If-None-Match or If-Modified-Since. While it's still
> small, that cuts the size approximately in half, and would allow you
> to skip even more processing if nothing has changed.
I'll look at that.
>
>
> > It updates this every time it does a git fetch, currently every
> 5 minutes.
>
> That up-to-five-minute delay, on top of whatever cronjob delay one has
> on one's animals, seems kind of sad. I've gotten kind of spoiled
> maybe
> by seeing first buildfarm results typically within 15 minutes of a
> push.
> But if we're trying to improve matters in this area, this doesn't seem
> like quite the way to go.
>
> But it does seem like this eliminates one expense. Now that you have
> that bit, maybe we could arrange a webhook or something that allows
> branches_of_interest.json to get updated immediately after a push?
>
>
> Webhooks are definitely a lot easier to implement in between our
> servers yeah, so that shouldn't be too hard. We could use the same
> hooks that we use for borka to build the docs, but have it just run
> whatever script it is the buildfarm needs. I assume it's just
> something trivial to run there, Andrew?
Yes, I think much better between servers. Currently the cron job looks
something like this:
*/5 * * * * cd $HOME/postgresql.git && git fetch -q &&
$HOME/website/bin/branches_of_interest.pl
That script is what sets up the json files.
I know nothing about git webhooks though, someone will have to point me
in the right direction.
I can set that up for you -- we have ready-made packages for 95% of what's needed for that one as we use it elsewhere in the infra. So I'll just set something up that will run that exact script (as the correct user of course) and comment out the cronjob,and then send you the details of what is set up where (I don't recall it offhand, but as it's the same we have elsewhere I'll find it quickly once I look into it).
On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander <magnus@hagander.net> wrote:
On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan <andrew@dunslane.net> wrote:
On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
> n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Andrew Dunstan <andrew@dunslane.net> writes:
b> > The buildfarm server now creates a companion to
> branches_of_interest.txt
> > called branches_of_interest.json which looks like this:
>
> ... okay ...
>
>
> Yeah, it's not as efficient as something like long polling or web
> sockets, but it is most definitely a lot simpler!
>
> If we're going to have a lot of animals do pulls of this file every
> minute or more, it's certainly a lot better to pull this small file
> than to make multiple git calls.
>
> It could trivially be made even more efficient by making the request
> with either a If-None-Match or If-Modified-Since. While it's still
> small, that cuts the size approximately in half, and would allow you
> to skip even more processing if nothing has changed.
I'll look at that.
>
>
> > It updates this every time it does a git fetch, currently every
> 5 minutes.
>
> That up-to-five-minute delay, on top of whatever cronjob delay one has
> on one's animals, seems kind of sad. I've gotten kind of spoiled
> maybe
> by seeing first buildfarm results typically within 15 minutes of a
> push.
> But if we're trying to improve matters in this area, this doesn't seem
> like quite the way to go.
>
> But it does seem like this eliminates one expense. Now that you have
> that bit, maybe we could arrange a webhook or something that allows
> branches_of_interest.json to get updated immediately after a push?
>
>
> Webhooks are definitely a lot easier to implement in between our
> servers yeah, so that shouldn't be too hard. We could use the same
> hooks that we use for borka to build the docs, but have it just run
> whatever script it is the buildfarm needs. I assume it's just
> something trivial to run there, Andrew?
Yes, I think much better between servers. Currently the cron job looks
something like this:
*/5 * * * * cd $HOME/postgresql.git && git fetch -q &&
$HOME/website/bin/branches_of_interest.pl
That script is what sets up the json files.
I know nothing about git webhooks though, someone will have to point me
in the right direction.I can set that up for you -- we have ready-made packages for 95% of what's needed for that one as we use it elsewhere in the infra. So I'll just set something up that will run that exact script (as the correct user of course) and comment out the cronjob,and then send you the details of what is set up where (I don't recall it offhand, but as it's the same we have elsewhere I'll find it quickly once I look into it).
Hi!
This should now be set up, and Andrew has been sent the instructions for how to access that setup on the buildfarm server. So hopefully it will now be updating the buildfarm server side of things within a couple of seconds from a commit, and not do any speculative pulls. But we'll keep an extra eye on it for a bit of course, as it's entirely possible I got something worng :)
(This is only the part git -> bf server, of course, as that step doesn't need any client changes it was easier to do quickly)
//Magnus
On 2022-11-22 Tu 13:04, Magnus Hagander wrote: > > > On Tue, Nov 22, 2022 at 12:10 AM Magnus Hagander <magnus@hagander.net> > wrote: > > > > On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan > <andrew@dunslane.net> wrote: > > > On 2022-11-21 Mo 16:20, Magnus Hagander wrote: > > n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> > wrote: > > > > Andrew Dunstan <andrew@dunslane.net> writes: > b> > The buildfarm server now creates a companion to > > branches_of_interest.txt > > > called branches_of_interest.json which looks like this: > > > > ... okay ... > > > > > > Yeah, it's not as efficient as something like long polling > or web > > sockets, but it is most definitely a lot simpler! > > > > If we're going to have a lot of animals do pulls of this > file every > > minute or more, it's certainly a lot better to pull this > small file > > than to make multiple git calls. > > > > It could trivially be made even more efficient by making the > request > > with either a If-None-Match or If-Modified-Since. While it's > still > > small, that cuts the size approximately in half, and would > allow you > > to skip even more processing if nothing has changed. > > > I'll look at that. > > > > > > > > > It updates this every time it does a git fetch, > currently every > > 5 minutes. > > > > That up-to-five-minute delay, on top of whatever cronjob > delay one has > > on one's animals, seems kind of sad. I've gotten kind > of spoiled > > maybe > > by seeing first buildfarm results typically within 15 > minutes of a > > push. > > But if we're trying to improve matters in this area, > this doesn't seem > > like quite the way to go. > > > > But it does seem like this eliminates one expense. Now > that you have > > that bit, maybe we could arrange a webhook or something > that allows > > branches_of_interest.json to get updated immediately > after a push? > > > > > > Webhooks are definitely a lot easier to implement in between our > > servers yeah, so that shouldn't be too hard. We could use > the same > > hooks that we use for borka to build the docs, but have it > just run > > whatever script it is the buildfarm needs. I assume it's just > > something trivial to run there, Andrew? > > > Yes, I think much better between servers. Currently the cron > job looks > something like this: > > > */5 * * * * cd $HOME/postgresql.git && git fetch -q && > $HOME/website/bin/branches_of_interest.pl > <http://branches_of_interest.pl> > > > That script is what sets up the json files. > > > I know nothing about git webhooks though, someone will have to > point me > in the right direction. > > > I can set that up for you -- we have ready-made packages for 95% > of what's needed for that one as we use it elsewhere in the infra. > So I'll just set something up that will run that exact script (as > the correct user of course) and comment out the cronjob,and then > send you the details of what is set up where (I don't recall it > offhand, but as it's the same we have elsewhere I'll find it > quickly once I look into it). > > > > Hi! > > This should now be set up, and Andrew has been sent the instructions > for how to access that setup on the buildfarm server. So hopefully it > will now be updating the buildfarm server side of things within a > couple of seconds from a commit, and not do any speculative pulls. But > we'll keep an extra eye on it for a bit of course, as it's entirely > possible I got something worng :) > > (This is only the part git -> bf server, of course, as that step > doesn't need any client changes it was easier to do quickly) > > The server side appears to be working well. The new client side code is being tested on crake and working fine - the all-up-to-date case takes just a second or two, almost all of which is taken with getting the json file from the server. No git calls at all are done on the client in this case. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Andrew Dunstan <andrew@dunslane.net> writes:
> The new client side code is being tested on crake and working fine - the
> all-up-to-date case takes just a second or two, almost all of which is
> taken with getting the json file from the server. No git calls at all
> are done on the client in this case.
Nice! I installed the new run_branches.pl file on sifaka, and it seems to
be doing the right things. With the much lower overhead, I've reduced
that cronjob's cycle time from five minutes to one, so that machine's
response time should be even better.
It'll probably help my slower animals even more, so I'm off to
update them as well.
regards, tom lane
Hi, On 2022-11-22 17:35:12 -0500, Andrew Dunstan wrote: > The server side appears to be working well. > > The new client side code is being tested on crake and working fine - the > all-up-to-date case takes just a second or two, almost all of which is > taken with getting the json file from the server. No git calls at all > are done on the client in this case. It's a huge improvement here. I start one set of animals via systemd timers and updated that buildfarm client to 24a6bb0 (because it shows the cpu/network resources used): Before: Nov 21 20:36:01 bf-valgrind-v4 systemd[1]: Starting PG buildfarm spinlock... ... Nov 21 20:36:14 bf-valgrind-v4 systemd[1]: bf@spinlock.service: Consumed 2.346s CPU time, received 578.3K IP traffic, sent32.3K IP traffic. Now: Nov 23 00:59:25 bf-valgrind-v4 systemd[1]: Starting PG buildfarm spinlock... ... Nov 23 00:59:26 bf-valgrind-v4 systemd[1]: bf@spinlock.service: Consumed 173ms CPU time, received 5.2K IP traffic, sent 1.8KIP traffic. Both of these are for builds that didn't do anything. Leaving wall clock time and resource usage aside, the output is also just a lot more readable: Nov 21 20:36:02 bf-valgrind-v4 run_branches.pl[1188989]: Mon Nov 21 20:36:02 2022: buildfarm run for francolin:REL_11_STABLEstarting Nov 21 20:36:02 bf-valgrind-v4 run_branches.pl[1188989]: francolin:REL_11_STABLE [20:36:02] checking out source ... Nov 21 20:36:04 bf-valgrind-v4 run_branches.pl[1188989]: francolin:REL_11_STABLE [20:36:04] checking if build run needed... Nov 21 20:36:04 bf-valgrind-v4 run_branches.pl[1188989]: francolin:REL_11_STABLE [20:36:04] No build required: last status= Sat Nov 19 21:56:55 2022 GMT, current snapshot = Sat Nov 19 20:36:52 2022 GMT, changed files = 0 Nov 21 20:36:04 bf-valgrind-v4 run_branches.pl[1189119]: Mon Nov 21 20:36:04 2022: buildfarm run for francolin:REL_12_STABLEstarting Nov 21 20:36:04 bf-valgrind-v4 run_branches.pl[1189119]: francolin:REL_12_STABLE [20:36:04] checking out source ... Nov 21 20:36:06 bf-valgrind-v4 run_branches.pl[1189119]: francolin:REL_12_STABLE [20:36:06] checking if build run needed... Nov 21 20:36:06 bf-valgrind-v4 run_branches.pl[1189119]: francolin:REL_12_STABLE [20:36:06] No build required: last status= Sat Nov 19 22:52:54 2022 GMT, current snapshot = Sat Nov 19 20:36:48 2022 GMT, changed files = 0 Nov 21 20:36:06 bf-valgrind-v4 run_branches.pl[1189233]: Mon Nov 21 20:36:06 2022: buildfarm run for francolin:REL_13_STABLEstarting Nov 21 20:36:06 bf-valgrind-v4 run_branches.pl[1189233]: francolin:REL_13_STABLE [20:36:06] checking out source ... Nov 21 20:36:08 bf-valgrind-v4 run_branches.pl[1189233]: francolin:REL_13_STABLE [20:36:08] checking if build run needed... Nov 21 20:36:08 bf-valgrind-v4 run_branches.pl[1189233]: francolin:REL_13_STABLE [20:36:08] No build required: last status= Sat Nov 19 23:12:55 2022 GMT, current snapshot = Sat Nov 19 20:36:33 2022 GMT, changed files = 0 Nov 21 20:36:08 bf-valgrind-v4 run_branches.pl[1189298]: Mon Nov 21 20:36:08 2022: buildfarm run for francolin:REL_14_STABLEstarting Nov 21 20:36:08 bf-valgrind-v4 run_branches.pl[1189298]: francolin:REL_14_STABLE [20:36:08] checking out source ... Nov 21 20:36:10 bf-valgrind-v4 run_branches.pl[1189298]: francolin:REL_14_STABLE [20:36:10] checking if build run needed... Nov 21 20:36:10 bf-valgrind-v4 run_branches.pl[1189298]: francolin:REL_14_STABLE [20:36:10] No build required: last status= Mon Nov 21 15:51:38 2022 GMT, current snapshot = Mon Nov 21 15:50:50 2022 GMT, changed files = 0 Nov 21 20:36:10 bf-valgrind-v4 run_branches.pl[1189364]: Mon Nov 21 20:36:10 2022: buildfarm run for francolin:REL_15_STABLEstarting Nov 21 20:36:10 bf-valgrind-v4 run_branches.pl[1189364]: francolin:REL_15_STABLE [20:36:10] checking out source ... Nov 21 20:36:12 bf-valgrind-v4 run_branches.pl[1189364]: francolin:REL_15_STABLE [20:36:12] checking if build run needed... Nov 21 20:36:12 bf-valgrind-v4 run_branches.pl[1189364]: francolin:REL_15_STABLE [20:36:12] No build required: last status= Mon Nov 21 16:26:28 2022 GMT, current snapshot = Mon Nov 21 15:50:50 2022 GMT, changed files = 0 Nov 21 20:36:12 bf-valgrind-v4 run_branches.pl[1189432]: Mon Nov 21 20:36:12 2022: buildfarm run for francolin:HEAD starting Nov 21 20:36:12 bf-valgrind-v4 run_branches.pl[1189432]: francolin:HEAD [20:36:12] checking out source ... Nov 21 20:36:14 bf-valgrind-v4 run_branches.pl[1189432]: francolin:HEAD [20:36:14] checking if build run needed... Nov 21 20:36:14 bf-valgrind-v4 run_branches.pl[1189432]: francolin:HEAD [20:36:14] No build required: last status= Mon Nov 21 17:31:31 2022 GMT, current snapshot = Mon Nov 21 16:59:29 2022 GMT, changed files = 0 vs Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:REL_11_STABLE is up to date. Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:REL_12_STABLE is up to date. Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:REL_13_STABLE is up to date. Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:REL_14_STABLE is up to date. Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:REL_15_STABLE is up to date. Nov 23 00:59:26 bf-valgrind-v4 run_branches.pl[4125973]: Wed Nov 23 00:59:26 2022: francolin:HEAD is up to date. Thanks a lot! Andres Freund
On Wed, Nov 23, 2022 at 2:09 PM Andres Freund <andres@anarazel.de> wrote: > It's a huge improvement here. Same here. eelpout + elver looking good, just a fraction of a second hitting that web server each minute. Long polling will be better and shave off 30 seconds (+/- 30) on start time, but this avoids a lot of useless churn without even needing a local mirror. Thanks Andrew!
On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro <thomas.munro@gmail.com> wrote:
On Wed, Nov 23, 2022 at 2:09 PM Andres Freund <andres@anarazel.de> wrote:
> It's a huge improvement here.
Same here. eelpout + elver looking good, just a fraction of a second
hitting that web server each minute. Long polling will be better and
shave off 30 seconds (+/- 30) on start time, but this avoids a lot of
useless churn without even needing a local mirror. Thanks Andrew!
Are you saying you still think it's worth pursuing longpoll or similar methods for it, or that this is good enough?
On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander <magnus@hagander.net> wrote: > On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro <thomas.munro@gmail.com> wrote: > Are you saying you still think it's worth pursuing longpoll or similar methods for it, or that this is good enough? I personally think it'd be pretty neat, to squeeze out that last bit of latency. Maybe it's overkill... The best idea I have so far for how to make it work more like a service but still require nothing more than cron (so it's not hard for people on systems where they don't even have root) is to have it start if not already running (current lock file scheme already does that) AND if some file buildfarm_enabled exists, or buildfarm_disabled doesn't exist or something like that, and then keep running while that's true. So if you need to turn it off for a while you can just touch/rm that, but normally it'll keep running its wait loop forever, and start up soon after a reboot; maybe it also exits if you touch the config file so it can restart next time and reread it, or something like that. Then it can spend all day in a loop that does 120s long polls, and start builds within seconds of a new commit landing. Curious to know how you'd build the server side. You mentioned a commit hook notifying some kind of long poll distributor. Would you use a Twisted/async/whatever-based server that knows how to handle lots of sockets efficiently, or just use old school web server tech that would block waiting for NOTIFY or something like that? You'd probably get away with that for the small numbers of animals involved (I mean, a couple of hundred web server threads/processes just sitting there waiting would be borderline acceptable I guess). But it'd be more fun to do it with async magic.
Thomas Munro <thomas.munro@gmail.com> writes:
> On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander <magnus@hagander.net> wrote:
>> On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>> Are you saying you still think it's worth pursuing longpoll or similar methods for it, or that this is good enough?
> I personally think it'd be pretty neat, to squeeze out that last bit
> of latency. Maybe it's overkill...
I can't get excited about pursuing the last ~30 seconds of delay
for launching tasks that are going to run 10 or 20 or more minutes
(where the future trend of those numbers is surely up not down).
The thing that was really significantly relevant here IMO was to
reduce the load on the central server, and I think we've done that.
Would adding longpoll reduce that further? In principle maybe,
but I'm not sure we have enough animals to make it worthwhile.
regards, tom lane
On 2022-11-23 We 16:59, Tom Lane wrote: > Thomas Munro <thomas.munro@gmail.com> writes: >> On Thu, Nov 24, 2022 at 10:00 AM Magnus Hagander <magnus@hagander.net> wrote: >>> On Wed, Nov 23, 2022 at 9:15 AM Thomas Munro <thomas.munro@gmail.com> wrote: >>> Are you saying you still think it's worth pursuing longpoll or similar methods for it, or that this is good enough? >> I personally think it'd be pretty neat, to squeeze out that last bit >> of latency. Maybe it's overkill... > I can't get excited about pursuing the last ~30 seconds of delay > for launching tasks that are going to run 10 or 20 or more minutes > (where the future trend of those numbers is surely up not down). > > The thing that was really significantly relevant here IMO was to > reduce the load on the central server, and I think we've done that. > Would adding longpoll reduce that further? In principle maybe, > but I'm not sure we have enough animals to make it worthwhile. > > Yeah, that's my feeling. We have managed to get a large improvement with a fairly small effort, I'm much less excited about getting another small improvement from a large effort. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com