Perhaps https://travis-ci.org/ or something similar could be used for this. That avoids any issues about random code.
That doesn't achieve any platform coverage, which is the main point here.
I don't think platform coverage is the first thing to worry about with patches, nor with ongoing development.
I think we are talking about solving two different problems...
The biggest win we'd get from something like Travis would be if the commitfest monitored for new patch files coming in for monitored threads and it created a new branch, applied the patches, and if they applied without error commit the branch and push to let Travis do it's thing. We wouldn't want that running in the main git repo, but it should be fine in a fork that's dedicated to that purpose.
Travis specifically would not help us with this, due to the dependency on gifhub, but something that knows how to run "patch ... && configure && make && make check" in a container would.
I'm unsure what would be easiest -- have something drive a "throwaway github repo" off the data in the CF app and try to pull things from there, or to just spawn containers and run it directly without travis.
The bigger issue with those is the usual -- how do you handle patches that have dependencies on each other,because they're always going to show up as broken individually. I guess we could tell people doing those to just push a git branch on github and register that one in the CF app (which does have some very basic support for tracking that, but I doubt anybody uses it today).
If the travis build failed, commitfest could notify the author.
It could also rebase master into each branch on a daily basis so authors would know very quickly if something got committed that broke their patch.
It could at least verify that the patch still applies, yes.
Obviously that doesn't remove the need for manual testing or the buildfarm, but it would at least let everyone know that the patch passed a smoke test
It's definitely something that would be useful, but as you say, it solves a different problem.
