Thread: BUG #16274: Repeated Libraries in Mac

The following bug has been logged on the website:

Bug reference:      16274
Logged by:          Krishnanunni R
Email address:      krishnanunnir97@gmail.com
PostgreSQL version: 11.7
Operating system:   Mac Os Catalina Version 10.15.1
Description:

On running the md5 command in the pgsql/lib/ folder using the following
command,
ind . -name "*.dylib" -exec md5 {} \; | sort -g -k 4
I am getting this as the output
MD5 (./libcrypto.1.1.dylib) = ebb2ebe266a6453b0cbd8b44a7fb64c5
MD5 (./libcrypto.dylib) = f2a42014ed6737d117c4732afffa783c
MD5 (./libecpg.6.11.dylib) = eda61536e715b3d1465f1a607def4004
MD5 (./libecpg.6.dylib) = eda61536e715b3d1465f1a607def4004
MD5 (./libecpg.dylib) = eda61536e715b3d1465f1a607def4004
MD5 (./libecpg_compat.3.11.dylib) = ece0b4117325a390d878dcc0d5733c51
MD5 (./libecpg_compat.3.dylib) = ece0b4117325a390d878dcc0d5733c51
MD5 (./libecpg_compat.dylib) = ece0b4117325a390d878dcc0d5733c51
MD5 (./libicuuc.57.1.dylib) = f2e3828d3c08e93ce45879d9025c90e6
MD5 (./libicuuc.57.dylib) = f2e3828d3c08e93ce45879d9025c90e6
MD5 (./libicuuc.dylib) = f2e3828d3c08e93ce45879d9025c90e6
MD5 (./libssl.1.1.dylib) = fdd4db7edc5df82640b795c0d18d9864
MD5 (./libuuid.1.1.dylib) = dd0416e32867556706f4156aa08490f8
MD5 (./libz.1.2.11.dylib) = ecde7cc9a6e9adcad042543c8de9e038
MD5 (./libz.1.dylib) = ecde7cc9a6e9adcad042543c8de9e038
MD5 (./libz.dylib) = ecde7cc9a6e9adcad042543c8de9e038
MD5 (./libssl.dylib) = 0c6f51a2ae9a8c75399a08de81b443db
MD5 (./libxslt.1.dylib) = 4a3c602e73ba982d19bde59913c749f4
MD5 (./libxslt.dylib) = 4a3c602e73ba982d19bde59913c749f4
MD5 (./libpq.5.11.dylib) = 9ae8352f4ee08ce42e08f34da45deb86
MD5 (./libpq.5.dylib) = 9ae8352f4ee08ce42e08f34da45deb86
MD5 (./libpq.dylib) = 9ae8352f4ee08ce42e08f34da45deb86
MD5 (./libicudata.57.1.dylib) = 34cfb3a54ceec04fb389069cfb78759d
MD5 (./libicudata.57.dylib) = 34cfb3a54ceec04fb389069cfb78759d
MD5 (./libicudata.dylib) = 34cfb3a54ceec04fb389069cfb78759d
MD5 (./libxml2.2.dylib) = 61b821c42a100d767e266dc18fbc5b45
MD5 (./libxml2.dylib) = 61b821c42a100d767e266dc18fbc5b45
MD5 (./libpgtypes.3.11.dylib) = 9848d4eec1c817440770ed6483fd126d
MD5 (./libpgtypes.3.dylib) = 9848d4eec1c817440770ed6483fd126d
MD5 (./libpgtypes.dylib) = 9848d4eec1c817440770ed6483fd126d
MD5 (./libicui18n.57.1.dylib) = 52398354f9930749d988a1253e61c1b1
MD5 (./libicui18n.57.dylib) = 52398354f9930749d988a1253e61c1b1
MD5 (./libicui18n.dylib) = 52398354f9930749d988a1253e61c1b1
MD5 (./libintl.9.dylib) = 5e57dc47461999ff90f404d0255e45c4
MD5 (./libintl.dylib) = 5e57dc47461999ff90f404d0255e45c4
MD5 (./libedit.0.dylib) = 83e829550c591270e97e469255bc4add
MD5 (./libedit.dylib) = 83e829550c591270e97e469255bc4add
The md5 value for many of the library files are identical. Does it imply all
these files are identical?
If so shouldn't it be removed?

> On 24 Feb 2020, at 13:12, PG Bug reporting form <noreply@postgresql.org> wrote:

> On running the md5 command in the pgsql/lib/ folder using the following
> command,
> ind . -name "*.dylib" -exec md5 {} \; | sort -g -k 4
> I am getting this as the output
> MD5 (./libcrypto.1.1.dylib) = ebb2ebe266a6453b0cbd8b44a7fb64c5
> MD5 (./libcrypto.dylib) = f2a42014ed6737d117c4732afffa783c
> MD5 (./libecpg.6.11.dylib) = eda61536e715b3d1465f1a607def4004
> MD5 (./libecpg.6.dylib) = eda61536e715b3d1465f1a607def4004
> MD5 (./libecpg.dylib) = eda61536e715b3d1465f1a607def4004

If you inspect the directory I think you will find that these are links to a
single file.  Use "find -type f" instead to deduplicate your file check.

cheers ./daniel

Daniel Gustafsson <daniel@yesql.se> writes:
>> On 24 Feb 2020, at 13:12, PG Bug reporting form <noreply@postgresql.org> wrote:
>> On running the md5 command in the pgsql/lib/ folder using the following
>> command,
>> ind . -name "*.dylib" -exec md5 {} \; | sort -g -k 4
>> I am getting this as the output
>> MD5 (./libcrypto.1.1.dylib) = ebb2ebe266a6453b0cbd8b44a7fb64c5
>> MD5 (./libcrypto.dylib) = f2a42014ed6737d117c4732afffa783c
>> MD5 (./libecpg.6.11.dylib) = eda61536e715b3d1465f1a607def4004
>> MD5 (./libecpg.6.dylib) = eda61536e715b3d1465f1a607def4004
>> MD5 (./libecpg.dylib) = eda61536e715b3d1465f1a607def4004

> If you inspect the directory I think you will find that these are links to a
> single file.  Use "find -type f" instead to deduplicate your file check.

Indeed.  The more interesting question is why libcrypto.1.1.dylib and
libcrypto.dylib appear *not* to be duplicates.  The latter should be
a symlink to the former, but seemingly it is not.

            regards, tom lane

> On 25 Feb 2020, at 07:19, Krishnanunni Radhakrishnan <krishnanunnir97@gmail.com> wrote:

Please avoid top-posting and keep the list on CC so everyone can learn from the
discussion.

> I have checked and they are neither hardlinks nor softlinks. Using Use "find -type f" returns the same output.
> An ls doesn't show any symbolic links, also the files have the same size. I thought it might possibly be a hardlink
soI tried checking the inode value for the files, they are also different. 
> I am facing this issue in binaries for  Mac -  version 11.7(http://sbp.enterprisedb.com/getfile.jsp?fileid=12377 ).

Thanks for clarifying that, it does explain why the files don't show up linked
as expected from a standard "make install", but not why the installer creates
multiple copies of the same file.  That does sound like a question for the EDB
installers team though.

cheers ./daniel

> On 24 Feb 2020, at 16:04, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Daniel Gustafsson <daniel@yesql.se> writes:
>>> On 24 Feb 2020, at 13:12, PG Bug reporting form <noreply@postgresql.org> wrote:
>>> On running the md5 command in the pgsql/lib/ folder using the following
>>> command,
>>> ind . -name "*.dylib" -exec md5 {} \; | sort -g -k 4
>>> I am getting this as the output
>>> MD5 (./libcrypto.1.1.dylib) = ebb2ebe266a6453b0cbd8b44a7fb64c5
>>> MD5 (./libcrypto.dylib) = f2a42014ed6737d117c4732afffa783c
>>> MD5 (./libecpg.6.11.dylib) = eda61536e715b3d1465f1a607def4004
>>> MD5 (./libecpg.6.dylib) = eda61536e715b3d1465f1a607def4004
>>> MD5 (./libecpg.dylib) = eda61536e715b3d1465f1a607def4004
>
>> If you inspect the directory I think you will find that these are links to a
>> single file.  Use "find -type f" instead to deduplicate your file check.
>
> Indeed.  The more interesting question is why libcrypto.1.1.dylib and
> libcrypto.dylib appear *not* to be duplicates.  The latter should be
> a symlink to the former, but seemingly it is not.

As was revealed in the thread now, the files stem from an EDB installer archive
and upon inspecting the archive I can confirm both that the symlinks are indeed
files as well as the differences in the libcrypto dylibs.

libssl.dylib and libss.1.1.dylib are also different, with both linking against
libcrypto.1.1.dylib The symbol tables are equal, but a binary diff does reveal
a set of differences between the files (as the md5 had already indicated).

I wonder if this could be artifact of how the produced binaries are built
against OpenSSL?  Not sure what would cause that though, but it's worrying to
see unexplained differences in shipped crypto libs.

cheers ./daniel

> On 26 Feb 2020, at 09:16, Krishnanunni Radhakrishnan <krishnanunnir97@gmail.com> wrote:
>
> Something else that came to my observation is there is no symbolic linkage between postmaster and postgres in mac
binariesfor version 11.7. They do have the same values for md5 

If it's not a link then it's at least good that they have the same md5 hash.
That does sound like another question for the installer team at EDB.

cheers ./daniel

> On 28 Feb 2020, at 14:13, Sandeep Thakkar <sandeep.thakkar@enterprisedb.com> wrote:
>
> Hi,
>
> I'll check why they are not symbolic links and fix the issue. The MD5 between them (libssl.dylib and libss.1.1.dylib
orlibcrypto.dylib and libcrypto.1.1.dylib) must be different as we codesign each file in the app bundle. But, anyways
thiswon't happen if there are symbolic links. Will check and revert. 

Thanks for following-up!

cheers ./daniel