Re: More efficient build farm animal wakeup? - Mailing list pgsql-hackers

From Magnus Hagander
Subject Re: More efficient build farm animal wakeup?
Date
Msg-id CABUevExYnZdJ+sCxK8TKni_dY19dnVdCbKYBMGLO-HNHjR+iGw@mail.gmail.com
Whole thread Raw
In response to Re: More efficient build farm animal wakeup?  (Tom Lane <tgl@sss.pgh.pa.us>)
Responses Re: More efficient build farm animal wakeup?
List pgsql-hackers
n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
Andrew Dunstan <andrew@dunslane.net> writes:
> The buildfarm server now creates a companion to branches_of_interest.txt
> called branches_of_interest.json which looks like this:

... okay ...

Yeah, it's not as efficient as something like long polling or web sockets, but it is most definitely a lot simpler!

If we're going to have a lot of animals do pulls of this file every minute or more, it's certainly a lot better to pull this small file than to make multiple git calls.

It could trivially be made even more efficient by making the request with either a If-None-Match or If-Modified-Since. While it's still small, that cuts the size approximately in half, and would allow you to skip even more processing if nothing has changed.


> It updates this every time it does a git fetch, currently every 5 minutes.

That up-to-five-minute delay, on top of whatever cronjob delay one has
on one's animals, seems kind of sad.  I've gotten kind of spoiled maybe
by seeing first buildfarm results typically within 15 minutes of a push.
But if we're trying to improve matters in this area, this doesn't seem
like quite the way to go.

But it does seem like this eliminates one expense.  Now that you have
that bit, maybe we could arrange a webhook or something that allows
branches_of_interest.json to get updated immediately after a push?

Webhooks are definitely a lot easier to implement in between our servers yeah, so that shouldn't be too hard. We could use the same hooks that we use for borka to build the docs, but have it just run whatever script it is the buildfarm needs. I assume it's just something trivial to run there, Andrew?

//Magnus

pgsql-hackers by date:

Previous
From: Robert Haas
Date:
Subject: Re: Re: Damage control for planner's get_actual_variable_endpoint() runaway
Next
From: Magnus Hagander
Date:
Subject: Re: More efficient build farm animal wakeup?