Thread: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

On 2018-May-30, REIX, Tony wrote:

> 32bit: 2 failures:
>    ============== running regression test queries ==============
>    test jsonb_plperl ... FAILED
>    test jsonb_plperlu ... FAILED
> 
> Expected/Result logs attached to this email.

This is not particularly helpful.  Please send the "regression.diff"
file instead.

> What about tests on Linux on i686 ?

Seems to work fine there, see lapwing:

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=lapwing&dt=2018-05-30%2017%3A20%3A01&stg=contrib-install-check-C

also dromedary:

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=dromedary&dt=2018-05-30%2016%3A47%3A23&stg=contrib-install-check-C

It's pretty obvious that the transform is broken on your platform.

Probably related to breakage fixed in 331b2369c0ad.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> It's pretty obvious that the transform is broken on your platform.

Seems so, but why?  The code involved doesn't look very machine-dependent.

I'm wondering about some sort of version skew or misinstallation on
the Perl side --- say, header files that we're using to compile
that don't match the libperl.so used at runtime.

            regards, tom lane

On Wed, May 30, 2018 at 03:36:27PM -0400, Tom Lane wrote:
> I'm wondering about some sort of version skew or misinstallation on
> the Perl side --- say, header files that we're using to compile
> that don't match the libperl.so used at runtime.

Tony, was the compiler used gcc or xlc?  Support for xlc 13 still needs
some work.  Here are two other, similar threads:

https://www.postgresql.org/message-id/B37989F2852398498001550C29155BE51790B48A@FRCRPVV9EX3MSX.ww931.my-it-solutions.net
https://www.postgresql.org/message-id/2fc764c7-2222-83e3-45c2-33c17853e13f@atos.net
--
Michael

Hi Michael,

We have used GCC, version 8.1 .
We also can build it with XLC, though I did not tried yet with v11beta1.

I know about the 2 threads you have provided.
Today, with our own patches, we are able to compile and run 100% OK PostgreSQL v10 or v9 on AIX 6 or AIX7, with GCC and
XLC.
(I'll submit these patches for AIX once we have run the stress/perf campaign we plan to do on AIX & Linux on P9)

I'm using nearly exactly the same .spec and .patch files for v10.4 and v11beta1.
However, I see that there are new files dealing with json coming with v11beta1:
# find $BUILD/postgresql-11beta1/64bit/ -name "json*" | wc
     72      72    7575
 # find $BUILD/postgresql-10.4/64bit/ -name "json*" | wc
     33      33    3180

Build/Tests in 64bit is 100% OK with v11beta1.

Regards

Cordialement,

Tony Reix

ATOS / Bull SAS
ATOS Expert
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : Michael Paquier [michael@paquier.xyz]
Envoyé : mercredi 30 mai 2018 22:25
À : Tom Lane
Cc : Alvaro Herrera; REIX, Tony; PostgreSQL-development; APEKE, SENA (ext); Peter Eisentraut; Anthony Bykov
Objet : Re: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

On Wed, May 30, 2018 at 03:36:27PM -0400, Tom Lane wrote:
> I'm wondering about some sort of version skew or misinstallation on
> the Perl side --- say, header files that we're using to compile
> that don't match the libperl.so used at runtime.

Tony, was the compiler used gcc or xlc?  Support for xlc 13 still needs
some work.  Here are two other, similar threads:
https://www.postgresql.org/message-id/B37989F2852398498001550C29155BE51790B48A@FRCRPVV9EX3MSX.ww931.my-it-solutions.net
https://www.postgresql.org/message-id/2fc764c7-2222-83e3-45c2-33c17853e13f@atos.net
--
Michael

Hi Álvaro

Here is the regression.diffs file.

Regards,

Cordialement,

Tony Reix

ATOS / Bull SAS
ATOS Expert
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : Alvaro Herrera [alvherre@2ndquadrant.com]
Envoyé : mercredi 30 mai 2018 20:46
À : REIX, Tony
Cc : PostgreSQL-development; APEKE, SENA (ext); Peter Eisentraut; Anthony Bykov
Objet : Re: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

On 2018-May-30, REIX, Tony wrote:

> 32bit: 2 failures:
>    ============== running regression test queries ==============
>    test jsonb_plperl ... FAILED
>    test jsonb_plperlu ... FAILED
>
> Expected/Result logs attached to this email.

This is not particularly helpful.  Please send the "regression.diff"
file instead.

> What about tests on Linux on i686 ?

Seems to work fine there, see lapwing:

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=lapwing&dt=2018-05-30%2017%3A20%3A01&stg=contrib-install-check-C

also dromedary:

https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=dromedary&dt=2018-05-30%2016%3A47%3A23&stg=contrib-install-check-C

It's pretty obvious that the transform is broken on your platform.

Probably related to breakage fixed in 331b2369c0ad.

--
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Hi Tom,

Hummm
About Perl, version 5.24.0 were installed 6 months ago and is still there.
And, still on the same AIX 7.2 machine, I've been able to build and test 100%OK version 10.4 before buiding/testing
v11beta1,and I've been able to build and test 100%OK version 9.6.9  after v11beta1. 

v11beta1 seems to add new 39 new files dealing with json.
For files: contrib/jsonb_plpython/jsonb_plpython.c and src/common/file_perm.c , I had to use   #define _LARGE_FILES 1
likeI did for 14 older files. That deals with lseek() and lseek64() definitions. 

Regards,

Cordialement,

Tony Reix

ATOS / Bull SAS
ATOS Expert
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : Tom Lane [tgl@sss.pgh.pa.us]
Envoyé : mercredi 30 mai 2018 21:36
À : Alvaro Herrera
Cc : REIX, Tony; PostgreSQL-development; APEKE, SENA (ext); Peter Eisentraut; Anthony Bykov
Objet : Re: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> It's pretty obvious that the transform is broken on your platform.

Seems so, but why?  The code involved doesn't look very machine-dependent.

I'm wondering about some sort of version skew or misinstallation on
the Perl side --- say, header files that we're using to compile
that don't match the libperl.so used at runtime.

                        regards, tom lane

"REIX, Tony" <tony.reix@atos.net> writes:
> For files: contrib/jsonb_plpython/jsonb_plpython.c and src/common/file_perm.c , I had to use   #define _LARGE_FILES 1
 like I did for 14 older files. That deals with lseek() and lseek64() definitions. 

I'm not following this.  Doesn't configure manage to figure out that
_LARGE_FILES=1 is needed?  It certainly tries to.

            regards, tom lane

"REIX, Tony" <tony.reix@atos.net> writes:
> It looks like configure does figure out that LARGE_FILES is required, only in 32bit.
> No need in 64bit.

Check ...

> However, in 32bit, though there is:
>    #define _LARGE_FILES 1
> in file :
>    src/include/pg_config.h
> I had to add it at the beg of file by means of a patch to several files:

This is surely not what's intended.  It seems like plpython is messing
things up somehow, probably through an #include-ordering error, but
I don't see the exact problem offhand.

I wondered why the existing 32-bit AIX buildfarm machines aren't showing
problems, but looking closer at them, they are manually forcing
_LARGE_FILES, which probably is masking things:

       'config_env' => {
                         'CC' => 'wrap-gcc -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix32',

Noah, why'd you do that, and would you be willing to remove it?  IMO
Postgres should work without that.

            regards, tom lane

Hi Tom,

Yes, there is something strange with _LARGE_FILES in 32bit.
However, that exists too with version 10.4 and 9.6.9 , and tests are 100% OK.
So, it seems that something new appears within v11

v11beta1 brings new json files. Either these files reveal some issue on AIX 32bit or they contain code that is not
compatiblewith AIX environment and some change should be applied... 


Cordialement,

Tony Reix

ATOS / Bull SAS
ATOS Expert
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : Tom Lane [tgl@sss.pgh.pa.us]
Envoyé : jeudi 31 mai 2018 16:28
À : REIX, Tony
Cc : Alvaro Herrera; PostgreSQL-development; APEKE, SENA (ext); Peter Eisentraut; Anthony Bykov; Noah Misch
Objet : Re: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

"REIX, Tony" <tony.reix@atos.net> writes:
> It looks like configure does figure out that LARGE_FILES is required, only in 32bit.
> No need in 64bit.

Check ...

> However, in 32bit, though there is:
>    #define _LARGE_FILES 1
> in file :
>    src/include/pg_config.h
> I had to add it at the beg of file by means of a patch to several files:

This is surely not what's intended.  It seems like plpython is messing
things up somehow, probably through an #include-ordering error, but
I don't see the exact problem offhand.

I wondered why the existing 32-bit AIX buildfarm machines aren't showing
problems, but looking closer at them, they are manually forcing
_LARGE_FILES, which probably is masking things:

       'config_env' => {
                         'CC' => 'wrap-gcc -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix32',

Noah, why'd you do that, and would you be willing to remove it?  IMO
Postgres should work without that.

                        regards, tom lane

"REIX, Tony" <tony.reix@atos.net> writes:
> v11beta1 brings new json files. Either these files reveal some issue on AIX 32bit or they contain code that is not
compatiblewith AIX environment and some change should be applied... 

One thing I notice is that jsonb_plperl.c starts with

#include "postgres.h"

#include "plpython.h"
#include "plpy_elog.h"
#include "plpy_typeio.h"
#include "utils/jsonb.h"
#include "utils/fmgrprotos.h"
#include "utils/numeric.h"

which does not follow the comment in plpython.h:

 * Include order should be: postgres.h, other postgres headers, plpython.h,
 * other plpython headers

So it seems this should be

#include "postgres.h"

#include "utils/jsonb.h"
#include "utils/fmgrprotos.h"
#include "utils/numeric.h"

#include "plpython.h"
#include "plpy_elog.h"
#include "plpy_typeio.h"

However, I don't see how that relates to _LARGE_FILES.  As long as
postgres.h is first, which it is, seems like that should work.

            regards, tom lane

On Thu, May 31, 2018 at 10:28:12AM -0400, Tom Lane wrote:
> "REIX, Tony" <tony.reix@atos.net> writes:
> > It looks like configure does figure out that LARGE_FILES is required, only in 32bit.
> > No need in 64bit.
> 
> Check ...
> 
> > However, in 32bit, though there is:
> >    #define _LARGE_FILES 1
> > in file :
> >    src/include/pg_config.h
> > I had to add it at the beg of file by means of a patch to several files:
> 
> This is surely not what's intended.  It seems like plpython is messing
> things up somehow, probably through an #include-ordering error, but
> I don't see the exact problem offhand.
> 
> I wondered why the existing 32-bit AIX buildfarm machines aren't showing
> problems, but looking closer at them, they are manually forcing
> _LARGE_FILES, which probably is masking things:
> 
>        'config_env' => {
>                          'CC' => 'wrap-gcc -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix32',
> 
> Noah, why'd you do that, and would you be willing to remove it?  IMO
> Postgres should work without that.

I did that to work around a problem like the one articulated upthread.
Specifically, a 64-bit build w/ plpython failed:

xlc_r -qnoansialias -g -O2 -qmaxmem=16384 -qsrcmsg -qlonglong  -I. -I. -I/home/nm/sw/python3-64/include/python3.4m
-I../../../src/include  -c -o plpy_elog.o plpy_elog.c
 
"/usr/include/unistd.h", line 201.17: 1506-343 (S) Redeclaration of lseek64 differs from previous declaration on line
199of "/usr/include/unistd.h".
 
"/usr/include/unistd.h", line 201.17: 1506-050 (I) Return type "long long" in redeclaration is not compatible with the
previousreturn type "long".
 
"/usr/include/unistd.h", line 201.17: 1506-377 (I) The type "long long" of parameter 2 differs from the previous type
"long".
"/usr/include/sys/lockf.h", line 64.20: 1506-343 (S) Redeclaration of lockf64 differs from previous declaration on line
62of "/usr/include/sys/lockf.h".
 
"/usr/include/sys/lockf.h", line 64.20: 1506-377 (I) The type "long long" of parameter 3 differs from the previous type
"long".
... <more like that> ...

Today's "configure" test concludes that we don't need _LARGE_FILES, because
off_t is 64-bit ("long", specifically) in this configuration.  The trouble
arises when Python.h does cause _LARGE_FILES to be defined.  Some system
headers don't tolerate a mix of _LARGE_FILES values in one compilation.  At
the time I added the workaround, I scratched down these candidates for a
proper fix:

1. Add "configure" test to determine whether Python defines _LARGE_FILES.
   When it does, define it ourselves at the top of each Python-associated
   source file.

2. Define _LARGE_FILES unconditionally on AIX.  That is, adopt the workaround
   as permanent.

3. Define _LARGE_FILES unconditionally.  This should be harmless, but I
   wouldn't tend to back-patch it.

4. Include system headers that react to _LARGE_FILES before including
   Python.h.  This is fragile; the list of affected headers may change.

I lean toward (2), because it will defend against other libraries choosing to
define _LARGE_FILES like Python does.  It would cause problems if some library
does an explicit "#undef _LARGE_FILES", but that's a less-likely choice.
Other preferences?

Noah Misch <noah@leadboat.com> writes:
> On Thu, May 31, 2018 at 10:28:12AM -0400, Tom Lane wrote:
>> I wondered why the existing 32-bit AIX buildfarm machines aren't showing
>> problems, but looking closer at them, they are manually forcing
>> _LARGE_FILES, which probably is masking things:
>>     'config_env' => {
>>     'CC' => 'wrap-gcc -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix32',
>> Noah, why'd you do that, and would you be willing to remove it?  IMO
>> Postgres should work without that.

> I did that to work around a problem like the one articulated upthread.
> Specifically, a 64-bit build w/ plpython failed:
> ...
> Today's "configure" test concludes that we don't need _LARGE_FILES, because
> off_t is 64-bit ("long", specifically) in this configuration.  The trouble
> arises when Python.h does cause _LARGE_FILES to be defined.

Ugh.  That's a pretty crummy decision on their part, although maybe there
was no better alternative.

This does not seem like it explains Tony's problem with AIX 32-bit,
though, as you'd think all concerned would agree _LARGE_FILES needs
to be 1 in that case.

> At the time I added the workaround, I scratched down these candidates for a
> proper fix:

> 1. Add "configure" test to determine whether Python defines _LARGE_FILES.
>    When it does, define it ourselves at the top of each Python-associated
>    source file.

That would make aspects of our extension ABI dependent on whether one had
configured with --enable-python, which would be surprising to say the
least.

> 2. Define _LARGE_FILES unconditionally on AIX.  That is, adopt the workaround
>    as permanent.

Perhaps.  I wonder though whether this is really an AIX-only problem.
(In particular, I wonder whether Python.h is clobbering _FILE_OFFSET_BITS
on other platforms.)  There's a comment in Autoconf's AC_SYS_LARGEFILE
that suggests it is, but ...

> 3. Define _LARGE_FILES unconditionally.  This should be harmless, but I
>    wouldn't tend to back-patch it.

It seems like variants of this issue should exist in all branches,
so I'm not really happy with taking a fix we're scared to back-patch.
If we were willing to do so, though, this might be OK.  Seems like there
are three possibilities:
* Defining _LARGE_FILES does something good, in which case we want it.
* Defining _LARGE_FILES does nothing.
* Defining _LARGE_FILES does something bad ... but it's hard to see
how that could be.

> 4. Include system headers that react to _LARGE_FILES before including
>    Python.h.  This is fragile; the list of affected headers may change.

Yeah, that seems fairly unworkable, though I wonder whether the
include-ordering advice in plpython.h isn't basically meant to achieve
this result.

We're still left with the question of why Tony is having a problem.
I wonder whether his build of Python.h is doing something strange
with _LARGE_FILES.

            regards, tom lane

On Thu, May 31, 2018 at 07:23:57PM -0400, Tom Lane wrote:
> Noah Misch <noah@leadboat.com> writes:
> > On Thu, May 31, 2018 at 10:28:12AM -0400, Tom Lane wrote:
> >> I wondered why the existing 32-bit AIX buildfarm machines aren't showing
> >> problems, but looking closer at them, they are manually forcing
> >> _LARGE_FILES, which probably is masking things:
> >>     'config_env' => {
> >>     'CC' => 'wrap-gcc -D_THREAD_SAFE=1 -D_LARGE_FILES=1 -maix32',
> >> Noah, why'd you do that, and would you be willing to remove it?  IMO
> >> Postgres should work without that.
> 
> > I did that to work around a problem like the one articulated upthread.
> > Specifically, a 64-bit build w/ plpython failed:
> > ...
> > Today's "configure" test concludes that we don't need _LARGE_FILES, because
> > off_t is 64-bit ("long", specifically) in this configuration.  The trouble
> > arises when Python.h does cause _LARGE_FILES to be defined.
> 
> Ugh.  That's a pretty crummy decision on their part, although maybe there
> was no better alternative.
> 
> This does not seem like it explains Tony's problem with AIX 32-bit,
> though, as you'd think all concerned would agree _LARGE_FILES needs
> to be 1 in that case.

Yep.  I suspect _LARGE_FILES is orthogonal to $SUBJECT, which looks like a
problem with floating point ABI or possibly structure layout.  Since, as you
say, all code concerned wants 64-bit off_t at all times, I doubt _LARGE_FILES
would be the cause of a structure layout mismatch.

> > At the time I added the workaround, I scratched down these candidates for a
> > proper fix:
> 
> > 1. Add "configure" test to determine whether Python defines _LARGE_FILES.
> >    When it does, define it ourselves at the top of each Python-associated
> >    source file.
> 
> That would make aspects of our extension ABI dependent on whether one had
> configured with --enable-python, which would be surprising to say the
> least.

I don't think it would.  We use _LARGE_FILES anyway on 32-bit.  On 64-bit,
_LARGE_FILES boils down to s/long/long long/, which is a C API change but not
an ABI change.

> > 2. Define _LARGE_FILES unconditionally on AIX.  That is, adopt the workaround
> >    as permanent.
> 
> Perhaps.  I wonder though whether this is really an AIX-only problem.
> (In particular, I wonder whether Python.h is clobbering _FILE_OFFSET_BITS
> on other platforms.)  There's a comment in Autoconf's AC_SYS_LARGEFILE
> that suggests it is, but ...
> 
> > 3. Define _LARGE_FILES unconditionally.  This should be harmless, but I
> >    wouldn't tend to back-patch it.
> 
> It seems like variants of this issue should exist in all branches,
> so I'm not really happy with taking a fix we're scared to back-patch.
> If we were willing to do so, though, this might be OK.  Seems like there
> are three possibilities:
> * Defining _LARGE_FILES does something good, in which case we want it.
> * Defining _LARGE_FILES does nothing.
> * Defining _LARGE_FILES does something bad ... but it's hard to see
> how that could be.

Fair.  I'd be content about back-patching for AIX, because 100% of AIX
buildfarm animals already do this.  While I don't anticipate specific breakage
on other platforms, testing the _LARGE_FILES interpretation of every 64-bit
libc on the planet feels like a recipe for surprises.

Hi Tom,

Here are information about the Perl version we are using on our AIX 7.2 machine.

However, as I said already, all PostgreSQL tests of v10.4 and 9.6.9 are OK, both 32 and 64bit.



# rpm -qa | grep perl
perl-5.24.0-3

This version 5.24.0 release 3 of Perl has been built on a AIX 6.1 machine (everything built on AIX 6.1 is compatible
withAIX 7). 

Looking at the log of the build/tests, I see:
32bit: Failed 1 test out of 2308, 99.96% okay.     t/porting/podcheck ............................................
FAILEDat test 1 
64bit: All tests successful.

We still build perl with XLC (v13.1.3) since we still have issues with GCC. Now trying to build 5.26.2 with GCC.


Running the following recommended command in 32 bit:
    setenv LIBPATH `pwd`:$LIBPATH; cd t; ./perl harness
I have:
Test Summary Report
-------------------
porting/checkcase.t                                             (Wstat: 0 Tests: 19966 Failed: 1)
  Failed test:  34
porting/podcheck.t                                              (Wstat: 0 Tests: 2741 Failed: 39)
  Failed tests:  1-2, 49, 64, 74, 170, 245, 247, 259-263
                321, 324, 412, 537, 539, 600, 744, 1095
                1104, 1139, 1151, 1178, 1180, 1187, 1196-1197
                1226, 1237, 1239, 1253-1254, 1263, 1265
                1289, 1483, 2741
../cpan/ExtUtils-Constant/t/Constant.t                          (Wstat: 0 Tests: 302 Failed: 3)
  Failed tests:  253, 255-256
Files=2410, Tests=835508, 1049 wallclock secs (42.12 usr 10.15 sys + 344.73 cusr 45.63 csys = 442.63 CPU)
Result: FAIL


About the libperl, I have:
# ar tv /opt/freeware/lib/perl5/5.24.0/ppc-aix-thread-multi/CORE/libperl.a
rwxr-xr-x     0/0     2245620 Nov 14 07:37 2017 libperl.o


About Perl header files, they are provided in 2 different directories:
64bit: /opt/freeware/lib/perl5/5.24.0/ppc-aix-thread-multi-64all/CORE/
32bit: /opt/freeware/lib/perl5/5.24.0/ppc-aix-thread-multi/CORE/

One file appears once: /opt/freeware/lib/perl5/5.24.0/Encode/encode.h , but is not used by PostgreSQL.

# diff /opt/freeware/lib/perl5/5.24.0/ppc-aix-thread-multi/CORE/perl.h
/opt/freeware/lib/perl5/5.24.0/ppc-aix-thread-multi-64all/CORE/perl.h
# identical

The Perl header files are coherent with libperl.a I think.

Cordialement,

Tony Reix

ATOS / Bull SAS
ATOS Expert
IBM Coop Architect & Technical Leader
Office : +33 (0) 4 76 29 72 67
1 rue de Provence - 38432 Échirolles - France
www.atos.net

________________________________________
De : Tom Lane [tgl@sss.pgh.pa.us]
Envoyé : mercredi 30 mai 2018 21:36
À : Alvaro Herrera
Cc : REIX, Tony; PostgreSQL-development; APEKE, SENA (ext); Peter Eisentraut; Anthony Bykov
Objet : Re: PostgreSQL 11 beta1 on AIX 7.2 : 2 failures in 32bit mode

Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> It's pretty obvious that the transform is broken on your platform.

Seems so, but why?  The code involved doesn't look very machine-dependent.

I'm wondering about some sort of version skew or misinstallation on
the Perl side --- say, header files that we're using to compile
that don't match the libperl.so used at runtime.

                        regards, tom lane