Thread: [psycopg] Build issue with Advance Toolchain

[psycopg] Build issue with Advance Toolchain

From

Devrim Gündüz

Date:

26 May 2017, 16:16:49

Hi,

I'm trying to compile psycpg2 on Power. For full optimization, I use Advance
Toolchain (AT), which is an extended version of GCC for Power.


The problem is: Parts of the code honors CC variable, part of does not.

Here is what I'm getting (see the difference between lines 1&2 and 3)

/opt/at10.0/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8 -D_GNU_SOURCE
-fPIC-fwrapv -O3 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4-grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8 -D_GNU_SOURCE -fPIC -fwrapv -O3 -fPIC
-DPSYCOPG_DEFAULT_PYDATETIME=1-DPSYCOPG_VERSION="2.7.1 (dt dec pq3 ext lo64)" -DPG_VERSION_NUM=90603 -DHAVE_LO64=1
-I/usr/include/python2.7-I. -I/usr/pgsql-9.6/include -I/usr/pgsql-9.6/include/server -c psycopg/microprotocols_proto.c
-obuild/temp.linux-ppc64le-2.7/psycopg/microprotocols_proto.o -Wdeclaration-after-statement 
/opt/at10.0/bin/gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8 -D_GNU_SOURCE
-fPIC-fwrapv -O3 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4-grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8 -D_GNU_SOURCE -fPIC -fwrapv -O3 -fPIC
-DPSYCOPG_DEFAULT_PYDATETIME=1-DPSYCOPG_VERSION="2.7.1 (dt dec pq3 ext lo64)" -DPG_VERSION_NUM=90603 -DHAVE_LO64=1
-I/usr/include/python2.7-I. -I/usr/pgsql-9.6/include -I/usr/pgsql-9.6/include/server -c psycopg/typecast.c -o
build/temp.linux-ppc64le-2.7/psycopg/typecast.o-Wdeclaration-after-statement 
gcc -pthread -shared -Wl,-z,relro build/temp.linux-ppc64le-2.7/psycopg/psycopgmodule.o
build/temp.linux-ppc64le-2.7/psycopg/green.obuild/temp.linux-ppc64le-2.7/psycopg/pqpath.o
build/temp.linux-ppc64le-2.7/psycopg/utils.obuild/temp.linux-ppc64le-2.7/psycopg/bytes_format.o
build/temp.linux-ppc64le-2.7/psycopg/libpq_support.obuild/temp.linux-ppc64le-2.7/psycopg/win32_support.o
build/temp.linux-ppc64le-2.7/psycopg/connection_int.obuild/temp.linux-ppc64le-2.7/psycopg/connection_type.o
build/temp.linux-ppc64le-2.7/psycopg/cursor_int.obuild/temp.linux-ppc64le-2.7/psycopg/cursor_type.o
build/temp.linux-ppc64le-2.7/psycopg/replication_connection_type.o
build/temp.linux-ppc64le-2.7/psycopg/replication_cursor_type.o
build/temp.linux-ppc64le-2.7/psycopg/replication_message_type.obuild/temp.linux-ppc64le-2.7/psycopg/diagnostics_type.o
build/temp.linux-ppc64le-2.7/psycopg/error_type.obuild/temp.linux-ppc64le-2.7/psycopg/lobject_int.o
build/temp.linux-ppc64le-2.7/psycopg/lobject_type.obuild/temp.linux-ppc64le-2.7/psycopg/notify_type.o
build/temp.linux-ppc64le-2.7/psycopg/xid_type.obuild/temp.linux-ppc64le-2.7/psycopg/adapter_asis.o
build/temp.linux-ppc64le-2.7/psycopg/adapter_binary.obuild/temp.linux-ppc64le-2.7/psycopg/adapter_datetime.o
build/temp.linux-ppc64le-2.7/psycopg/adapter_list.obuild/temp.linux-ppc64le-2.7/psycopg/adapter_pboolean.o
build/temp.linux-ppc64le-2.7/psycopg/adapter_pdecimal.obuild/temp.linux-ppc64le-2.7/psycopg/adapter_pint.o
build/temp.linux-ppc64le-2.7/psycopg/adapter_pfloat.obuild/temp.linux-ppc64le-2.7/psycopg/adapter_qstring.o
build/temp.linux-ppc64le-2.7/psycopg/microprotocols.obuild/temp.linux-ppc64le-2.7/psycopg/microprotocols_proto.o
build/temp.linux-ppc64le-2.7/psycopg/typecast.o-L. -L/usr/pgsql-9.6/lib -lpython2.7 -lpq -o
build/lib.linux-ppc64le-2.7/psycopg2/_psycopg.so
unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1

Can you please let me know how I can fix this?

Thanks!

Regards,
--
Devrim Gündüz
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Attachment

signature.asc

Re: [psycopg] Build issue with Advance Toolchain

From

Daniele Varrazzo

Date:

26 May 2017, 18:31:28

On Fri, May 26, 2017 at 11:16 AM, Devrim Gündüz <devrim@gunduz.org> wrote:

> The problem is: Parts of the code honors CC variable, part of does not.
>
> Here is what I'm getting (see the difference between lines 1&2 and 3)

Don't know anything specific about this toolchain. However the bad
step is the linking, so it seems it compiles ok but links bad.

Do `python-config` --cflags and --ldflags return a sensible output?

-- Daniele

Re: [psycopg] Build issue with Advance Toolchain

From

Adrian Klaver

Date:

26 May 2017, 19:02:51

On 05/26/2017 05:31 AM, Daniele Varrazzo wrote:
> On Fri, May 26, 2017 at 11:16 AM, Devrim Gündüz <devrim@gunduz.org> wrote:
>
>> The problem is: Parts of the code honors CC variable, part of does not.
>>
>> Here is what I'm getting (see the difference between lines 1&2 and 3)
>
> Don't know anything specific about this toolchain. However the bad
> step is the linking, so it seems it compiles ok but links bad.

What I see is that in compiling it uses:

/opt/at10.0/bin/gcc

but when it gets to the linking it does:

gcc

which leads to:

unable to execute gcc: No such file or directory
error: command 'gcc' failed with exit status 1

>
> Do `python-config` --cflags and --ldflags return a sensible output?
>
> -- Daniele
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

Re: [psycopg] Build issue with Advance Toolchain

From

Devrim Gündüz

Date:

26 May 2017, 20:22:24

Hi Daniele,

On Fri, 2017-05-26 at 13:31 +0100, Daniele Varrazzo wrote:
> Do `python-config` --cflags and --ldflags return a sensible output?

-bash-4.2$ python-config --cflags
-I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions-fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8
-D_GNU_SOURCE-fPIC -fwrapv -O3 -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mcpu=power7 -mtune=power8 -D_GNU_SOURCE
-fPIC-fwrapv -O3 
-bash-4.2$ python-config --ldflags
-lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic

FWIW, here is what I export in the spec file for all Power packages (atpath: /opt/at10.0)

        CFLAGS="${CFLAGS} $(echo %{__global_cflags} | sed 's/-O2/-O3/g') -m64 -mcpu=power8 -mtune=power8
-I%{atpath}/include"
        CXXFLAGS="${CXXFLAGS} $(echo %{__global_cflags} | sed 's/-O2/-O3/g') -m64 -mcpu=power8 -mtune=power8
-I%{atpath}/include"
        LDFLAGS="-L%{atpath}/%{_lib}"
        CC=%{atpath}/bin/gcc; export CC

Regards,
--
Devrim Gündüz
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Attachment

signature.asc

Re: [psycopg] Build issue with Advance Toolchain

From

Devrim Gündüz

Date:

26 May 2017, 20:24:12

Hi Adrian,

On Fri, 2017-05-26 at 06:02 -0700, Adrian Klaver wrote:
> What I see is that in compiling it uses:
>
> /opt/at10.0/bin/gcc
>
> but when it gets to the linking it does:
>
> gcc
>
> which leads to:
>
> unable to execute gcc: No such file or directory
> error: command 'gcc' failed with exit status 1

Ah, right. Can we override the gcc there?

Regards,
--
Devrim Gündüz
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

Attachment

signature.asc

Re: [psycopg] Build issue with Advance Toolchain

From

Adrian Klaver

Date:

26 May 2017, 20:30:39

On 05/26/2017 07:24 AM, Devrim Gündüz wrote:
>
> Hi Adrian,
>
> On Fri, 2017-05-26 at 06:02 -0700, Adrian Klaver wrote:
>> What I see is that in compiling it uses:
>>
>> /opt/at10.0/bin/gcc
>>
>> but when it gets to the linking it does:
>>
>> gcc
>>
>> which leads to:
>>
>> unable to execute gcc: No such file or directory
>> error: command 'gcc' failed with exit status 1
>
> Ah, right. Can we override the gcc there?

I searched on doing that with setup.py and quickly came to the
realization that I know a lot less about the Python build process then I
thought. This is going to be something that someone else, Daniele I'm
guessing, is going to have to answer I'm afraid.

>
> Regards,
>

--
Adrian Klaver
adrian.klaver@aklaver.com

Re: [psycopg] Build issue with Advance Toolchain

From

Daniele Varrazzo

Date:

26 May 2017, 22:23:03

On Fri, May 26, 2017 at 3:22 PM, Devrim Gündüz <devrim@gunduz.org> wrote:

> FWIW, here is what I export in the spec file for all Power packages (atpath: /opt/at10.0)
>
>         CFLAGS="${CFLAGS} $(echo %{__global_cflags} | sed 's/-O2/-O3/g') -m64 -mcpu=power8 -mtune=power8
-I%{atpath}/include"
>         CXXFLAGS="${CXXFLAGS} $(echo %{__global_cflags} | sed 's/-O2/-O3/g') -m64 -mcpu=power8 -mtune=power8
-I%{atpath}/include"
>         LDFLAGS="-L%{atpath}/%{_lib}"
>         CC=%{atpath}/bin/gcc; export CC

So it looks these settings affect the compile phase but not the built
phase, right? In particular CC...

Could it be a problem with either setuptools or distutils? At the top
of setup.py there is a stanza like:

try:
    from setuptools import setup, Extension
except ImportError:
    from distutils.core import setup, Extension

Could you try forcing one of the two, and see if it a bug in one of
these build tool?

Also see some answers here, I think they point towards some sort of
solution: <https://stackoverflow.com/questions/5967065/python-distutils-not-using-correct-version-of-gcc>.
Looking at that it seems "compiler_so" gets chosen wrongly. You could
take a look at 'distutils/ccompiler.py' adding some prints and some
breakpoints to see what it ends up being. Also notice that 'python
setup.py build takes a --compiler parameter: I don't know if it can be
used (but I think it only accepts a subset of what you can override
using $CC)

I'm afraid I'm a bit shotgunning a solution: I'm not extremely
experienced with these tools either...

-- Daniele

Re: [psycopg] Build issue with Advance Toolchain

From

Adrian Klaver

Date:

26 May 2017, 22:39:56

On 05/26/2017 07:24 AM, Devrim Gündüz wrote:
>
> Hi Adrian,
>
> On Fri, 2017-05-26 at 06:02 -0700, Adrian Klaver wrote:
>> What I see is that in compiling it uses:
>>
>> /opt/at10.0/bin/gcc
>>
>> but when it gets to the linking it does:
>>
>> gcc
>>
>> which leads to:
>>
>> unable to execute gcc: No such file or directory
>> error: command 'gcc' failed with exit status 1
>
> Ah, right. Can we override the gcc there?

Have you looked at this:


https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W51a7ffcf4dfd_4b40_9d82_446ebc23c550/page/IBM%20Advance%20Toolchain%20for%20PowerLinux%20Documentation?section=usage

"After completing the installation steps, the Advance Toolchain is ready
for usage. Just call the program directly, for example /opt/atX.X/bin/gcc.

However, some applications have complex build systems (for example:
autotool, make, cmake) in which is necessary to correctly set the
environment PATH, for example:

     PATH=/opt/atX.X/bin:/opt/atX.X/sbin:$PATH make

On cmake build systems, it is also necessary to set the
CMAKE_PREFIX_PATH with AT's path, for example:

     CMAKE_PREFIX_PATH=/opt/atX.X/

"


>
> Regards,
>


--
Adrian Klaver
adrian.klaver@aklaver.com