Adding CI to our tree - Mailing list pgsql-hackers

Hi,

For several development efforts I found it to be incredibly valuable to push
changes to a personal repository and see a while later whether tests succeed
on a number of different platforms.  This is especially useful for platforms
that are quite different from ones own platform, like e.g. windows in my case.

Of course everybody can set this up for themselves. However, doing so well is
a significant effort, particularly if windows is to be supported well. And
doubly so if useful things like getting backtraces for crashes is desirable
([1])

We do a form of pre-commit CI via cfbot. That is valuable. But it's not really
comparable to having CI in tree - one need to post to the list and one cannot
adjust the dependencies etc installed for the CI runs.


New contributors (and quite a bit of older ones too) IMO expect to be able to
see whether their changes work as-is, without sending a patch to the list.


An obvious criticism of the effort to put CI runner infrastructure into core
is that they are effectively all proprietary technology, and that we should be
hesistant to depend too much on one of them. I think that's a valid
concern. However, once one CI integration is done, a good chunk (but not all!)
the work is transferrable to another CI solution, which I do think reduces the
dependency sufficiently.


The attached patch adds CI using cirrus-ci. The reason for choosing cirrus
were that
a) Thomas has ended up using cirrus for cfbot
b) cirrus provides a comparatively wide variety of operating systems
c) it allows custom VM images to be used.
d) it does not require a login to look at

c) is very valuable to be able to test e.g. upcoming linux versions,
pre-installing software on systems that do not support docker (freebsd), and
being faster to boot once the image is more than a trivial size.  I've created
a number of images for testing of the aio patchset [2]


Right now the patch attached
- runs check-world on FreeBSD, Linux, macOS - all using gcc
  - freebsd, linux use a custom generated image
  - macOS installs missing dependencies at runtime, with some caching
  - all use ccache to make subsequent compilation faster
- runs all the tests I could find on windows, via vcregress.pl
- checks for compiler warnings on linux, with both clang and gcc

- captures all logs after a failing run
- generates backtraces from core files (the output format differs between platforms)
- allows to limit CI to certain OSs, by adding
  ci-os-only: (freebsd|linux|macos|windows)+ to the commit message
  (useful when fixing a platform dependent problem)

Example output of a
- successful run: https://cirrus-ci.com/build/4625606928236544
- github interface for the same: https://github.com/anarazel/postgres/runs/3772435617
- failed run on windows, with backtrace: https://cirrus-ci.com/task/6640561307254784?logs=cat_dumps#L150

Comments? Rotten tomatoes?


There's some polishing we should do before actually adding this to the
tree. But I wanted to discuss the idea before investing even more time.

One policy discussion that we'd have to have is who should control the images
used for CI. Right now that's on my personal google cloud account - which I am
happy to do, but medium - long term that'd not be optimal.


Thanks to Thomas - I based this on hist .cirrus.yml file. And made several
contributions later. Also thanks to Andrew, who helped out with some windows
issues I hit at some point...

Greetings,

Andres Freund

[1] I did get this to work on windows, but it does require a small set of
changes to work reliably, I'll start a separate thread about it.
[2] https://github.com/anarazel/pg-vm-images/