Thread: Installing via pip under Win7 + virtualenv + VC++2008

On 06/02/2011 07:07 PM, Stephen Lacy wrote:
> But, when running python, I get this:
>
> Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit
> (Intel)] on
> win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import psycopg2
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "psycopg2\__init__.py", line 71, in <module>
>     from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
> ImportError: DLL load failed: The specified module could not be found.
>>>>
>
> Any debugging help would be greatly appreciated.  I know I can install
> win-psycopg2, but I'd rather just get the pip install working for
> everyone, since it automates the install in a scriptable way.

The first thing I'd do is

import sys
print sys.path

and take a look at the list of dirs.  On Windows XP, non-virtualenv, I
see Python\lib, Python\DLLs, Python\lib\plat-win,
Python\lib\site-packages, among others.

Joe

On Fri, Jun 3, 2011 at 12:07 AM, Stephen Lacy <slacy@slacy.com> wrote:

> ImportError: DLL load failed: The specified module could not be found.

the file _psycopg.pyd is a dll. For its relatively small size it is
probably dynamically linked to the libpq.dll. I think the system
either fails to find libpq.dll or some of the other dependencies (they
are listed in the setup.py).

I suggest you to use a tool to check its dll dependencies (maybe
http://www.dependencywalker.com/ could be ok) and see if there is
anything missing. If libpq.dll is the missing one, try putting it in a
PATH directory or in the psycopg2 package directory.

Jason, what is the default for building psycopg on windows: static or
dynamic libpq? Is there any default setting we may change to make
psycopg installable with pip/easy_install (if it currently fails - I
don't know).

-- Daniele

On Fri, Jun 3, 2011 at 5:50 PM, Stephen Lacy <slacy@slacy.com> wrote:
> Yeah, I went down this track as well.  And in short, here's where I am:
>
> I didn't have C:\...\PostgreSQL\8.4\bin on my path.
> So, after building my env and installing, I added it to my path.  "import
> psycopg2" worked, which was good.
> But then, as a sanity check, I nuked the env and am now trying to rebuild
> it.  "pip install psycopg" is currently failing with "Error: pg_config
> executable not found" even though I've got it on my path and can run it
> directly from the commandline.  Even more odd is how "pip install psycopg2"
> works *without* PostgreSQL in my path, but *fails* if I have it in.  Very,
> very confusing. :)
> Here's the output from "pip install psycopg2" when I have PostgreSQL in my
> path.
[...]

I have the feeling (and not only from this post) that pip makes a bit
of a mess with psycopg. It is another whole layer of magic that can
fail: could you please test without it? "python setup.py build_ext"
should be sufficient to reproduce all the problems.

Apart from that, the function autodetect_pg_config_path_windows() in
setup.py looks _really_ messy... After looking at that function, I
don't have the heart to blame pip anymore!

Unfortunately, while I can't believe it takes a whopping 76 lines of
python to find where pg_config is, I don't have a windows installation
to test with. I would just place a breakpoint and follow step by step
what it does, and where it fails.

It may be time of course to clean up some of the crusts setup.py has
accumulated in 10-odds years: many "features" are there to deal with
conditions we don't meet anymore (e.g. we now require pg_config and
refuse guesswork, there are still checks for python < 2.4 etc).

-- Daniele

On Fri, Jun 3, 2011 at 6:57 PM, Stephen Lacy <slacy@slacy.com> wrote:
> Great, thanks.  I'm doing this now, and may just loop in jericso via github.
>  I suppose the recommended way of submitting a patch is to clone jericso's
> psycopg2 repo on github and submit a pull request?

I think Jason will want to catch up with my devel branch, so cloning
it would be ok.

I add a note: I am a little bit intimidated by the setup.py because I
don't know exactly the extent at which it has to produce its work: I
don't know for example what compilers we want to support. But it's
probably about time to make some decisions in order to cut dead
branches away. For me it would be ok to clean up the setup in order to
make it work with the sanest build configurations (in terms of
compiler used and way of linking libraries), and add back any of the
hacks only if/when required. Let's just decide what makes sense to
support. For this kind of decisions, posting on the list would be
perfect.

Thank you.

-- Daniele

On Fri, 3 Jun 2011, Daniele Varrazzo wrote:

> On Fri, Jun 3, 2011 at 12:07 AM, Stephen Lacy <slacy@slacy.com> wrote:
>
>> ImportError: DLL load failed: The specified module could not be found.
>
> the file _psycopg.pyd is a dll. For its relatively small size it is
> probably dynamically linked to the libpq.dll. I think the system
> either fails to find libpq.dll or some of the other dependencies (they
> are listed in the setup.py).
>
> I suggest you to use a tool to check its dll dependencies (maybe
> http://www.dependencywalker.com/ could be ok) and see if there is
> anything missing. If libpq.dll is the missing one, try putting it in a
> PATH directory or in the psycopg2 package directory.
>
> Jason, what is the default for building psycopg on windows: static or
> dynamic libpq? Is there any default setting we may change to make
> psycopg installable with pip/easy_install (if it currently fails - I
> don't know).

The default for bulding psycopg is static, but...
it is probably being linked to the dynamic library if the prebuilt
version of PostgreSQL is used.


The main difference between static and dynamic is the link library name.

When built from source, the static library is named libpq.lib and the
dynamic library is libpqdll.lib.

When using the binary distrubtion of PostgreSQL for windows, the dynamic
library is named libpq.lib (I am not aware of a static library in the
distribution).  Note that using the binary distribution will require not
only the libpq.dll in the path, but also the openssl libraries in the path
(libeay32.dll and ssleay32.dll).


Don't ask me why the name difference between source build and binary
distribution (besides to cause confusion). :)


I am probably a special case as I build everything from source.  I can see
most people building psycopg2 from source using the prebuilt PostgreSQL
binaries, so we should make sure it builds in that configuration.
Challenges I see is that 1) we should make sure that the libpq dlls are
in the path so can be found with python, 2) if a psycopg2 distribution
file is made, we would also need to include in the dlls.


-jason

>
> -- Daniele
>

On Sat, Jun 4, 2011 at 4:19 PM, Jason Erickson <jerickso@stickpeople.com> wrote:

> I am probably a special case as I build everything from source.  I can see
> most people building psycopg2 from source using the prebuilt PostgreSQL
> binaries, so we should make sure it builds in that configuration. Challenges
> I see is that 1) we should make sure that the libpq dlls are in the path so
> can be found with python, 2) if a psycopg2 distribution file is made, we
> would also need to include in the dlls.

I see these relevant scenarios of people building psycopg on windows:

1) You use gcc to build python extension with prebuilt PG binaries
(e.g. with a package such as
http://www.develer.com/oss/GccWinBinaries)

2) You use VC with prebuilt PG binaries

3) You are Jason, providing fine binaries since the 20th century :)

these are in ascending order of importance for me (also because people
in 1 are probably used enough to tweak small glitches in the setup).

Stephen is already happily hacking on the setup in a github branch: he
has already cleaned it up a little bit. Once he says he is fine, you
may test if everything is ok for you and then release, so scenarios 2
and 3 are covered.

I may try later and check if the build works on win/gcc too on an old
vm of mine, but I don't consider it a precondition for release.

Thank you both for your help!

-- Daniele

On Sat, Jun 4, 2011 at 4:47 PM, Daniele Varrazzo
<daniele.varrazzo@gmail.com> wrote:

> 1) You use gcc to build python extension with prebuilt PG binaries

I have given a go at building psycopg with mingw. I've made a couple
of patches, one about dropping warnings, another to support Python 3
in autodetect_pg_config_path_windows(). Stephen, you may want to merge
it as it is in the area you are hacking on.

It works good enough, but as Jason says it builds a pyd depending on
the libpq.dll, which in turn depends on 4-5 dlls found in the
Postgres/bin directory. So the product still requires some care to be
deployed, but if the static lib is not installed by the postgres
package I guess there's little that can be done. An option would be to
copy all the dlls in the psycopg2 dir... would it be ugly?

Another problem I've had was caused by the problem described in
http://bugs.python.org/issue3308 due to an old msvcr90.lib I had in my
build system. With the buggy lib it wasn't possible to build for
Python 2.6 and 3.1.

Both the missing localtime in msvcr90.lib and the dll dependencies
have been easy to debug using the tool at
http://www.dependencywalker.com/ - without it the error messages are
painfully unhelpful.

Apart from the patches already committed in my devel, I think I'm fine
with the scenario mingw/win. I will see to add a web page about the
tools used.

-- Daniele