Thread: Koji, mock, fedpkg, etc?
Hi folks I've been looking into ways to streamline packaging, as I'm going to need to be rolling on-demand packages soon and want to stay as close as possible to PGDG. In the process, I've noticed that the PGDG yum/rpm packaging team seems to have a lot of homebrew infrastructure that duplicates facilities provided by Fedora/RH. For example, the use of Jenkins for package building instead of koji and mock. I've been taking the same approach as PGDG to date, as I only just became aware of Koji and mock, as well as the ancillary tools like the koji command line client, rpkg/fedpkg, etc. Have these tools been specifically looked at? I'm wondering if they might make package building/maintenance easier. My only concern with Koji is that it doesn't seem friendly toward low-demand dynamic package building, where VMs for package building are launched on-demand and shut down after use. OTOH, I might simply not have noticed a relevant tool or plugin yet. I thought I'd ask here and solicit opinions/experience, in case someone's already looked into this toolset and concluded that it doesn't meet PGDG packaging needs. Or, if it's unfamiliar, find out whether it might actually help solve any current challenges with maintenance. Comments? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Hi, On Mon, 2014-07-28 at 12:36 +0800, Craig Ringer wrote: > I've been looking into ways to streamline packaging, as I'm going to > need to be rolling on-demand packages soon and want to stay as close as > possible to PGDG. > > In the process, I've noticed that the PGDG yum/rpm packaging team seems > to have a lot of homebrew infrastructure that duplicates facilities > provided by Fedora/RH. For example, the use of Jenkins for package > building instead of koji and mock. We don't use any of the automated tools. Right now, there is *almost* zero automation. At 2007, when I started the yum repository project, I and Darcy (Buskermolen) tried to install koji a few times, but honestly we could not succeed. Then, gave up. > I've been taking the same approach as PGDG to date, as I only just > became aware of Koji and mock, as well as the ancillary tools like the > koji command line client, rpkg/fedpkg, etc. I really love koji -- at least when I use it for Fedora and EPEL packaging. > Have these tools been specifically looked at? I'm wondering if they > might make package building/maintenance easier. TBH, I am not that eager to use automated tools -- at least the signing process needs manual work. Still, I am open to ideas of course. > My only concern with Koji is that it doesn't seem friendly toward > low-demand dynamic package building, where VMs for package building are > launched on-demand and shut down after use. OTOH, I might simply not > have noticed a relevant tool or plugin yet. FWIW, our VMs are never shut down, and IIRC koji can work with it. > I thought I'd ask here and solicit opinions/experience, in case > someone's already looked into this toolset and concluded that it doesn't > meet PGDG packaging needs. Or, if it's unfamiliar, find out whether it > might actually help solve any current challenges with maintenance. Well, I think Jeff should also say his opinion. So far, I am pretty happy with what we have now, since I am used to it -- which also means that I may not be aware of the abnormalities of the process. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/29/2014 04:31 AM, Devrim Gündüz wrote: > > Hi, > > On Mon, 2014-07-28 at 12:36 +0800, Craig Ringer wrote: > >> I've been looking into ways to streamline packaging, as I'm going >> to need to be rolling on-demand packages soon and want to stay as >> close as possible to PGDG. >> >> In the process, I've noticed that the PGDG yum/rpm packaging team >> seems to have a lot of homebrew infrastructure that duplicates >> facilities provided by Fedora/RH. For example, the use of Jenkins >> for package building instead of koji and mock. > > We don't use any of the automated tools. Right now, there is > *almost* zero automation. OK, that's informative. You use a bunch of generated Jenkins build jobs with dependencies to do builds, if the wiki docs are correct. Is that still true? If so, you might be interested in some recent patches I've submitted to Jenkins - some reliability fixes for automanaged AWS EC2 build workers, and a change that makes it easy to set the build status to "unstable" with a shell return code. > At 2007, when I started the yum repository project, I and Darcy > (Buskermolen) tried to install koji a few times, but honestly we > could not succeed. Then, gave up. Given the rate at which that stuff changes and the way the docs tend to go from non-existent to bitrotted in no time, I'm not shocked. >> I've been taking the same approach as PGDG to date, as I only >> just became aware of Koji and mock, as well as the ancillary >> tools like the koji command line client, rpkg/fedpkg, etc. > > I really love koji -- at least when I use it for Fedora and EPEL > packaging. Good to know. I'm trying to figure out if it's worth my setting it up for in-house package builds, but as the current PGDG tree doesn't seem to be set up to follow its conventions I think it might not yet be the way to go. In particular, the svn tree with subdirs per distro seems to be quite different to the koji convention of a git tree with branches per distro. > TBH, I am not that eager to use automated tools -- at least the > signing process needs manual work. Still, I am open to ideas of > course. RPM signing should be easily automated except for the actual input of a keysigning passphrase or enabling of the key. >> My only concern with Koji is that it doesn't seem friendly >> toward low-demand dynamic package building, where VMs for package >> building are launched on-demand and shut down after use. OTOH, I >> might simply not have noticed a relevant tool or plugin yet. > > FWIW, our VMs are never shut down, and IIRC koji can work with it. Makes sense. >> I thought I'd ask here and solicit opinions/experience, in case >> someone's already looked into this toolset and concluded that it >> doesn't meet PGDG packaging needs. Or, if it's unfamiliar, find >> out whether it might actually help solve any current challenges >> with maintenance. > > Well, I think Jeff should also say his opinion. So far, I am > pretty happy with what we have now, since I am used to it -- which > also means that I may not be aware of the abnormalities of the > process. I'm familiar with that state of affairs in other areas I work with. So I get it. It can also be frustrating to have new people then come diving in and saying "why don't you use Hot New Tool #42?". Now, speaking of Hot New Tool, have you looked at tito? Its multi-release, multi-package management looks great, and it supports direct integration with mock and koji. - -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT1uREAAoJELBXNkqjr+S2oKQH/j+UnXE7VXO6xyEeIZJrwvKq 7K60e51TpdusETdw9PYjoPnZUl88PtTifX7rHxdV1yItVpq98IHEBJUjwXUO12o3 FpTQBVqO/xpqgGelauOHOG0LJr54j6U6vG+j9lN8Iu/ASqxhKJ+W56tYn+0APuh2 T4c5W1qUm6a8gq/SvZoQ2UtmM72qTPYGobKWMvCGtHlc90SiB31bkNmQ+xmhpxmW B50bUQPHaHfS3jY9s/T8c/xNlnu3XM+wmsFxzpTSEqo8sJG2SLBEN8O3Xh8qPEZg oqp3LQxb9dAoVyYDczMRVqaWo6ZUjCAGaHR19/KJVejOpR6vguuP7nLRWd0k33s= =rRsb -----END PGP SIGNATURE-----
Hi, On Tue, 2014-07-29 at 08:01 +0800, Craig Ringer wrote: > You use a bunch of generated Jenkins build jobs with dependencies to > do builds, if the wiki docs are correct. Is that still true? Which wiki page is this? I can't find such text in http://wiki.pgrpms.org > Now, speaking of Hot New Tool, have you looked at tito? Its > multi-release, multi-package management looks great, and it supports > direct integration with mock and koji. No, I did not hear about it before. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR
Attachment
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/29/2014 01:14 PM, Devrim Gündüz wrote: > > Hi, > > On Tue, 2014-07-29 at 08:01 +0800, Craig Ringer wrote: >> You use a bunch of generated Jenkins build jobs with >> dependencies to do builds, if the wiki docs are correct. Is that >> still true? > > Which wiki page is this? I can't find such text in > http://wiki.pgrpms.org Ah, ignore me. It's the apt packaging that's done through Jenkins. https://wiki.postgresql.org/wiki/Apt/Jenkins >> Now, speaking of Hot New Tool, have you looked at tito? Its >> multi-release, multi-package management looks great, and it >> supports direct integration with mock and koji. > > No, I did not hear about it before. I just did, and suggest that you don't. It isn't pretty as it stands. Doesn't really validate configurations and report errors. Relies on poking a remote git repo all the time rather than working in a local tree. Invokes shell commands from python instead of using shutil. Makes lots of assumptions about how the tree is organized, how releases are tagged, doesn't break downstream patches into a patch series named by commit heading, etc. Not convinced it's worth pursuing. - -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJT2IA1AAoJELBXNkqjr+S28XoIALVP1VrQqI9eCoXX6DizPwfZ mWdncUTAaBhuMajEXFC+CjnkgarwRfC+N5pcxa4O736CFRgph/vAgIPI3okNglui I9/U9pNJlLePnkjq37Hwnkbrf2nLu1oAQNgWWT0wR0c1Xhw6BeueLA6xJHkpr+/n xPFweMuiXCe0Pkz82GRszBShIY8B4E0K8uaRhvJJmQsYlPAU5R9YoxY9gRq0cw65 rpfxfX4ZJMc3SGxNdVp0k8qMJzv6ihRD5eJQVJqS5yuQoIkoQuaXtrsxa0uiDMNC NUChEFbKmD7aqpUHPr8tDQar/2m+ezSwT0GNRx1ghWwSIu60shcZo5cbs0qvGTY= =7SA4 -----END PGP SIGNATURE-----