Is there a way to find out about new git commits that is more efficient and timely than running N git fetches or whatever every minute in a cron job? Maybe some kind of long polling where you send an HTTP request that says "I think the tips of branches x, y, z are at 111, 222, 333" and the server responds when that ceases to be true?
I'm not aware of any such thing standardized for git, but it wouldn't be hard to build one for that (I'm talking primarily about the server side here, not how to integrate that into the buildfarm side of things).
We could also set something up whereby we could fire off webhooks when branches change (easy enough for registered servers in the buildfarm as we can easily avoid abuse there -- it would take more work to make something like that a public service, due to the risk of abuse). But that may actually be worse off, since I bet a lot of buildfarm animals (most even?) are probably sitting behind a NAT gateway of some kind, meaning consuming webhooks is hard.
I did something similar for how we did things on borka (using some internal pginfra webhooks that are not available to the public at this time), but I had to revert that because of issues with concurrent buildfarm runs in the environment that we had set up. But we are using it for the snapshots docs builder, to make sure the website for that gets updated immediately after a commit on master. But the principle definitely work.
Another thing to consider would be that something like this would cause all buildfarm clients to start git pull:ing down changes at more or less exactly the same time. Though in total that would probably still mean a lot less load than those that "git pul" very frequently today, it could potentially lead to some nets with lots of bf clients experiencing some level of bandwidth filling or something. Could probably be solved pretty easily with a random delay (which doesn't have to be long, as for most git pulls it will be a very quick operation), just something that's worth considering.
tl,tr; it's not there now, but yes if we can find a smart way for th ebf clients to consume it, it is something we could build and deploy fairly easily.
