Re: More efficient build farm animal wakeup? - Mailing list pgsql-hackers

On Mon, Nov 21, 2022 at 11:42 PM Andrew Dunstan <andrew@dunslane.net> wrote:

On 2022-11-21 Mo 16:20, Magnus Hagander wrote:
> n Mon, Nov 21, 2022 at 9:58 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
>     Andrew Dunstan <andrew@dunslane.net> writes:
b>     > The buildfarm server now creates a companion to
>     branches_of_interest.txt
>     > called branches_of_interest.json which looks like this:
>
>     ... okay ...
>
>
> Yeah, it's not as efficient as something like long polling or web
> sockets, but it is most definitely a lot simpler!
>
> If we're going to have a lot of animals do pulls of this file every
> minute or more, it's certainly a lot better to pull this small file
> than to make multiple git calls.
>
> It could trivially be made even more efficient by making the request
> with either a If-None-Match or If-Modified-Since. While it's still
> small, that cuts the size approximately in half, and would allow you
> to skip even more processing if nothing has changed.


I'll look at that.


>
>
>     > It updates this every time it does a git fetch, currently every
>     5 minutes.
>
>     That up-to-five-minute delay, on top of whatever cronjob delay one has
>     on one's animals, seems kind of sad.  I've gotten kind of spoiled
>     maybe
>     by seeing first buildfarm results typically within 15 minutes of a
>     push.
>     But if we're trying to improve matters in this area, this doesn't seem
>     like quite the way to go.
>
>     But it does seem like this eliminates one expense.  Now that you have
>     that bit, maybe we could arrange a webhook or something that allows
>     branches_of_interest.json to get updated immediately after a push?
>
>
> Webhooks are definitely a lot easier to implement in between our
> servers yeah, so that shouldn't be too hard. We could use the same
> hooks that we use for borka to build the docs, but have it just run
> whatever script it is the buildfarm needs. I assume it's just
> something trivial to run there, Andrew?


Yes, I think much better between servers. Currently the cron job looks
something like this:


*/5 * * * * cd $HOME/postgresql.git && git fetch -q &&
$HOME/website/bin/branches_of_interest.pl


That script is what sets up the json files.


I know nothing about git webhooks though, someone will have to point me
in the right direction.

I can set that up for you -- we have ready-made packages for 95% of what's needed for that one as we use it elsewhere in the infra. So I'll just set something up that will run that exact script (as the correct user of course) and comment out the cronjob,and then send you the details of what is set up where (I don't recall it offhand, but as it's the same we have elsewhere I'll find it quickly once I look into it).