Thread: Linux Update Experience

We are running PGDG Postgres 9.6 and 12 on RHEL7.

 

Our Linux team does global Linux updates on a quarterly basis (yum update).

 

We are hitting more and more update problems.

 

Some troubles this time:

 

+ Postgis24 has been updated to Postgis30

+ Postgres 12.2 has been updated to Postgres 12.3 claiming missing requirements:

 

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)

           Requires: llvm-toolset-7-clang >= 4.0.1

 

Question: How to you handle your Linux update cycles? Not updating anymore?

 

Thanks,

Markus

Hi Markus,

we are using PGDG on Debian / Ubuntu trying to update Linux as frequent as possible. I do not remember many issues if any. I would not expect:

* problems between postgres minor version upgrades (12.2 -> 12.3). We have PG 12.3 & PostGIS 3.0 on Ubuntu Bionic and do not remember any update problems.

* major upgrades when keeping same Linux distribution version. Our older Ubuntu Xenial keeps PG 9.6 & PostGIS 2.3.

So from my perspective: linux updates should not trigger major DB upgrades (or postGis), minor DB upgrades should be smooth -> it is possible to update Linux continuously. Major DB upgrades have so be triggered manually (on same Linux distro version or newer). On that Ubuntu Bionic server, we started on PG10 upgrading DB together with PostGis every year with dump / restore procedure to fresh new DB.

--

Jiří Fejfar

On Thu, 2020-05-28 at 09:00 +0000, Marco Lechner wrote:
> Hi Markus,
>  
> at the moment we are facing similar conflicts on Oracle LInux 7 (wich
> is derived from RHEL) – we manage our machines using Spacewalk. The
> conflicts occur (as expected) on Spacewalk as well as on manually
> using yum:
>  
> Fehler: Paket: postgresql11-devel-11.8-1PGDG.rhel7.x86_64
> (oraclelinux7postgresql11)
>             Benötigt: llvm-toolset-7-clang >= 4.0.1

FWIW, postgresql-devel can't be updated on CentOS 7 currently either.
The 12.2 packages were fine but I have not been able to update to 12.3.

Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 (pg12)
           Requires: llvm-toolset-7-clang >= 4.0.1

That llvm-toolset-7-clang dependency is not present in the CentOS, EPEL
or PostgreSQL repos.

On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:
> We are running PGDG Postgres 9.6 and 12 on RHEL7.
> 
> Our Linux team does global Linux updates on a quarterly basis (yum update).
> 
> We are hitting more and more update problems.
> 
> Some troubles this time:
> 
> + Postgis24 has been updated to Postgis30
> 
> + Postgres 12.2 has been updated to Postgres 12.3 claiming missing 
> requirements:
> 
> Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64 
> (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
> 
>             Requires: llvm-toolset-7-clang >= 4.0.1
> 
> Question: How to you handle your Linux update cycles? Not updating anymore?

See here:

https://yum.postgresql.org/news-newreporpmsreleased.php

And if you have community account:

https://redmine.postgresql.org/issues/5483

To contact the RPM packagers directly:

https://yum.postgresql.org/contact.php

> 
> Thanks,
> 
> Markus
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

> -----Ursprüngliche Nachricht-----
> Von: Adrian Klaver <adrian.klaver@aklaver.com>
> Gesendet: Donnerstag, 28. Mai 2020 16:15
> An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch>; PostgreSQL General
> <pgsql-general@lists.postgresql.org>
> Betreff: Re: Linux Update Experience
>
> On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:
> > We are running PGDG Postgres 9.6 and 12 on RHEL7.
> >
> > Our Linux team does global Linux updates on a quarterly basis (yum update).
> >
> > We are hitting more and more update problems.
> >
> > Some troubles this time:
> >
> > + Postgis24 has been updated to Postgis30
> >
> > + Postgres 12.2 has been updated to Postgres 12.3 claiming missing
> > requirements:
> >
> > Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
> > (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
> >
> >             Requires: llvm-toolset-7-clang >= 4.0.1
> >
> > Question: How to you handle your Linux update cycles? Not updating anymore?
>
> See here:
>
> https://yum.postgresql.org/news-newreporpmsreleased.php
>
> And if you have community account:
>
> https://redmine.postgresql.org/issues/5483
>
> To contact the RPM packagers directly:
>
> https://yum.postgresql.org/contact.php
>
> >
> > Thanks,
> >
> > Markus
> >
>
>
> --
> Adrian Klaver
> adrian.klaver@aklaver.com
[Zwettler Markus (OIZ)]



Hi Adrian,

I'm not talking about this specific bug or its resolution.

I want to talk about the Linux update problem in general.

Anyone updating Linux might get such nerving dependency troubles.

How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

Cheers,
Markus

On 5/28/20 7:36 AM, Zwettler Markus (OIZ) wrote:
>> -----Ursprüngliche Nachricht-----
>> Von: Adrian Klaver <adrian.klaver@aklaver.com>
>> Gesendet: Donnerstag, 28. Mai 2020 16:15
>> An: Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch>; PostgreSQL General
>> <pgsql-general@lists.postgresql.org>
>> Betreff: Re: Linux Update Experience
>>
>> On 5/28/20 12:59 AM, Zwettler Markus (OIZ) wrote:
>>> We are running PGDG Postgres 9.6 and 12 on RHEL7.
>>>
>>> Our Linux team does global Linux updates on a quarterly basis (yum update).
>>>
>>> We are hitting more and more update problems.
>>>
>>> Some troubles this time:
>>>
>>> + Postgis24 has been updated to Postgis30
>>>
>>> + Postgres 12.2 has been updated to Postgres 12.3 claiming missing
>>> requirements:
>>>
>>> Error: Package: postgresql12-devel-12.3-1PGDG.rhel7.x86_64
>>> (imx_product_3rd_party_postgresql_repository_postgresql12_rhel7_x86_64)
>>>
>>>              Requires: llvm-toolset-7-clang >= 4.0.1
>>>
>>> Question: How to you handle your Linux update cycles? Not updating anymore?
>>
>> See here:
>>
>> https://yum.postgresql.org/news-newreporpmsreleased.php
>>
>> And if you have community account:
>>
>> https://redmine.postgresql.org/issues/5483
>>
>> To contact the RPM packagers directly:
>>
>> https://yum.postgresql.org/contact.php
>>
>>>
>>> Thanks,
>>>
>>> Markus
>>>
>>
>>
>> --
>> Adrian Klaver
>> adrian.klaver@aklaver.com
> [Zwettler Markus (OIZ)]
> 
> 
> 
> Hi Adrian,
> 
> I'm not talking about this specific bug or its resolution.
> 
> I want to talk about the Linux update problem in general.
> 
> Anyone updating Linux might get such nerving dependency troubles. >
> How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

If you are installing via packages and the package managers are doing 
there job then there should not be an issue. The dependencies should be 
taken care of. As to version changes, that depends on the software. For 
instance Postgres 12.2 --> 12.3 is a minor/bug release so it should be 
something you get. Not sure about the PostGIS upgrade, that probably 
depends on what repo(s) you are using.  The bottom line is if you ask 
for upgrades/updates you are going to get them. This means keeping track 
of what is happening with updates to your software. Now you can pin/lock 
versions, search on pinning/locking packages. The downside to that is 
missing important bug fixes. I would say not updating qualifies as a 
foot gun.

> 
> Cheers,
> Markus
> 
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

On Thu, May 28, 2020 at 02:36:34PM +0000, Zwettler Markus (OIZ) wrote:

> Hi Adrian,
>
> I'm not talking about this specific bug or its resolution.
>
> I want to talk about the Linux update problem in general.
>
> Anyone updating Linux might get such nerving dependency troubles.
>
> How do you handle this situation? Updating more frequently? Updating less frequently? Not updating anymore?

If we ask ourselves general questions there can't be, by the
very nature of the question, any more specific answer beyond:

It depends.

Conventional wisdom holds it that updating "more frequently"
(not "accruing technical debt") helps -- the trick is to find
the balance between early effort and lag.

Karsten
--
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B

Zwettler Markus (OIZ) <Markus.Zwettler@zuerich.ch> writes:

> Hi Marco,
>
>  
>
> How do you handle these conflicts? No longer updating that regularly or not at all anymore?
>

Not doing the updates is a poor option due to the potential security
vulnerabilities this may lead to. Likewise, delaying the application of
updates is going to increase risks as well. In fact, we have found such
approaches can make the situation worse. Delaying updates tends to
result in more updates being applied at once, which makes it harder to
identify problems when they do occur.

I think the best approach is to apply updates as soon as possible. Apply
the updates to a test or uat environment (or your dev environment if you
don't have a test/uat/staging one). If there are issues, resolve them
before applying the updates in prod.

We have found it rare for updates to be an issue if your running the
packages from the distribution. Problems seem to be more common when
your running packages sourced from an external repository which may lag
behind changes made by the distribution. When issues do occur, we look
at the type of update e.g. security, bug fix, bump in dependency
versions, new version etc and make a call as to the priority and respond
accordingly. This may mean delaying applying the update, actively
working to resolve the issue with investigations and debugging, raising
an issue/bug with the package maintainers etc.

We also classify all our systems, services, databases etc according to
whether they are core business processes or supporting processes. We
apply updates to supporting systems before core systems. This also
affects when we apply updates. For example, we would not apply updates
to a core system on Friday afternoon. In fact, we may apply updates to
core systems outside main business hours. If issues are encountered when
applying updates to core systems, resolution of those issues are highest
priority. For secondary systems, we are more likely to do the updates
during business hours, will accept longer outages/down times and may not
make resolution of issues the highest priority. 

On 2020-05-28 14:36:34 +0000, Zwettler Markus (OIZ) wrote:
> I'm not talking about this specific bug or its resolution.
>
> I want to talk about the Linux update problem in general.
>
> Anyone updating Linux might get such nerving dependency troubles.

In my experience (having administrated Linux servers for 25 years),
dependency troubles outside of major updates are rare. But of course if
you do that long enough with enough different systems you will run into
one sooner or later.

Some ways to minimize the frequency of such troubles:

* Use a base distribution with a lot of packages. RHEL has a lot fewer
  packages than Debian, so when we used RHEL (we still have a few
  servers) we used external repositories (RPMForge, EPEL, ...) a lot
  more. Some of those external repos may not be as well maintained as
  the base distro and of course they may not coordinate with each other.
  So more external repos mean a higher risk of conflicts.

* Use specialized systems. In the 90's servers were big and expensive,
  so we had few of them and each was running a lot of different
  services. Planning an upgrade took months. These days we use (mostly)
  VMs, each of which is running only one service. That greatly reduces
  the number or packages installed and the number of external repos,
  thus reducing the potential for conflicts.

* Update frequently. That reduces the risk of needing a package which
  has since been deleted from a repo, but more importantly it makes it
  easier to pinpoint the cause of a conflict.

When a conflict does occur. being familiar with the packaging system
helps a lot. Sometimes just uninstalling a few packages helps. Sometimes
something in a repo has changed and you need to change the configuration
to match (as has apparently happened here), so being on relevant
announce-lists of having the URL of the repo website handy helps.
Sometimes you can force installation (althought that will often cause
problems later). In some cases I built my own packages.

        hp


--
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

## Peter J. Holzer (hjp-pgsql@hjp.at):

> * Update frequently. That reduces the risk of needing a package which
>   has since been deleted from a repo, but more importantly it makes it
>   easier to pinpoint the cause of a conflict.

This. Plus: make sure you can re-create any machine in a fully deterministic
manner - that way, you can easily create a test environment to match
production (minus RAM/CPU/storage) for testing upgrades beforehand.

Rationale: experience shows that using Test as "first stage" and carrying
changes forward to Production results in a "contaminated" test environment;
before long, results of failed experiments have accumulated on Test,
Production and Test are diverging, and at that point Test has lost it's
purpose.
(For some people, that's a point for containerization: you don't change
a running container, but package a new one. Other environments have so
much Production with all the redundancy etc that they can "test in
production" and just scrap-and-replace failed tests, but that's not an
option if you have just a handful of systems.)

Regards,
Christoph

-- 
Spare Space