Thread: PIP Package Building for pgAdmin4

Hi

On Wed, Mar 9, 2016 at 2:24 PM, Muhammad Aqeel
<muhammad.aqeel@enterprisedb.com> wrote:
> Hi,
>
> PFA the patch to build PIP package for pgAdmin4 web component. It builds PIP
> wheel file based on Python version in PATH.
>
> This pip package build setup is supported for python version 2.6 and above.
> The pip package can easily be built with a make statement.
>
> Makefile has following targets.
>   - install-required: To install packages required to build PIP package.
>   - all: To build PIP package
>   - install-pip: To install built PIP package
>   - clean: To clean the sources.

Let's try to keep the Makefile target names specific to what we're
building, e.g.

install-pip-requirements: To install packages required to build PIP
package (not sure that's the best name though)
install-pip: To install built PIP package
pip: To build the PIP package

Eventually the "all"  target will likely build the runtime and maybe
other packages as well, so for now, it should probably look something
like:

all: install-pip-requirements pip

Similarly, "clean" will eventually do other things too, so we can
create "clean-pip", which can be called with other further clean
targets by "clean".

> setup_pgadmin4.py file contains all the Meta information needed to build the
> PIP package.

I think that should be moved to pkg/pip, where it can later live
alongside directories like win32, osx and rpm etc. Would setup_pip.py
be a better name as well?

> The version for the pip package is picked up from web/config.py file.

Urgh - don't try to parse that file from a Python script. It's a
Python script itself, so you can just import it - e.g.

import config
version = config.APP_VERSION

Thanks!

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

EBROKENPATCH

(pgadmin4)piranha:pgAdmin4 dpage$ git apply
~/Downloads/pgadmin4_pip_framework_v5.patch
/Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:41: trailing whitespace.
python pkg/pip/setup_pip.py bdist_wheel
/Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:45: trailing whitespace.
all: install-pip-requirements pip
/Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:47: trailing whitespace.
install-pip-requirements:
/Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:59: trailing whitespace.
pip:
/Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:73: trailing whitespace.
fi
error: Makefile: No such file or directory
error: pkg/pip/setup_pip.py: No such file or directory

On Fri, Mar 11, 2016 at 11:19 AM, Muhammad Aqeel
<muhammad.aqeel@enterprisedb.com> wrote:
> PFA the updated patch.
>
> On Wed, Mar 9, 2016 at 10:25 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi
>>
>> On Wed, Mar 9, 2016 at 2:24 PM, Muhammad Aqeel
>> <muhammad.aqeel@enterprisedb.com> wrote:
>> > Hi,
>> >
>> > PFA the patch to build PIP package for pgAdmin4 web component. It builds
>> > PIP
>> > wheel file based on Python version in PATH.
>> >
>> > This pip package build setup is supported for python version 2.6 and
>> > above.
>> > The pip package can easily be built with a make statement.
>> >
>> > Makefile has following targets.
>> >   - install-required: To install packages required to build PIP package.
>> >   - all: To build PIP package
>> >   - install-pip: To install built PIP package
>> >   - clean: To clean the sources.
>>
>> Let's try to keep the Makefile target names specific to what we're
>> building, e.g.
>>
>> install-pip-requirements: To install packages required to build PIP
>> package (not sure that's the best name though)
>> install-pip: To install built PIP package
>> pip: To build the PIP package
>
>
> Done.
>>
>>
>> Eventually the "all"  target will likely build the runtime and maybe
>> other packages as well, so for now, it should probably look something
>> like:
>>
>> all: install-pip-requirements pip
>
>
> Done.
>>
>>
>> Similarly, "clean" will eventually do other things too, so we can
>> create "clean-pip", which can be called with other further clean
>> targets by "clean".
>
>
> Done.
>
>>
>>
>> > setup_pgadmin4.py file contains all the Meta information needed to build
>> > the
>> > PIP package.
>>
>> I think that should be moved to pkg/pip, where it can later live
>> alongside directories like win32, osx and rpm etc. Would setup_pip.py
>> be a better name as well?
>
>
> Done.
>
>>
>>
>> > The version for the pip package is picked up from web/config.py file.
>>
>> Urgh - don't try to parse that file from a Python script. It's a
>> Python script itself, so you can just import it - e.g.
>>
>> import config
>> version = config.APP_VERSION
>
>
> Done.
>
>>
>>
>> Thanks!
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thanks - committed with some minor tweaks:

- Avoid packaging *.pyc and pgadmin4.db
- Add some more keywords
- Minor re-arrangement of setup_pip.py and the Makefile.

Thanks again!

On Mon, Mar 14, 2016 at 6:33 AM, Muhammad Aqeel
<muhammad.aqeel@enterprisedb.com> wrote:
> Sorry for broken patch.
> PFA the corrected patch.
>
> --Aqeel
>
> On Fri, Mar 11, 2016 at 5:51 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> EBROKENPATCH
>>
>> (pgadmin4)piranha:pgAdmin4 dpage$ git apply
>> ~/Downloads/pgadmin4_pip_framework_v5.patch
>> /Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:41: trailing
>> whitespace.
>> python pkg/pip/setup_pip.py bdist_wheel
>> /Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:45: trailing
>> whitespace.
>> all: install-pip-requirements pip
>> /Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:47: trailing
>> whitespace.
>> install-pip-requirements:
>> /Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:59: trailing
>> whitespace.
>> pip:
>> /Users/dpage/Downloads/pgadmin4_pip_framework_v5.patch:73: trailing
>> whitespace.
>> fi
>> error: Makefile: No such file or directory
>> error: pkg/pip/setup_pip.py: No such file or directory
>>
>> On Fri, Mar 11, 2016 at 11:19 AM, Muhammad Aqeel
>> <muhammad.aqeel@enterprisedb.com> wrote:
>> > PFA the updated patch.
>> >
>> > On Wed, Mar 9, 2016 at 10:25 PM, Dave Page <dpage@pgadmin.org> wrote:
>> >>
>> >> Hi
>> >>
>> >> On Wed, Mar 9, 2016 at 2:24 PM, Muhammad Aqeel
>> >> <muhammad.aqeel@enterprisedb.com> wrote:
>> >> > Hi,
>> >> >
>> >> > PFA the patch to build PIP package for pgAdmin4 web component. It
>> >> > builds
>> >> > PIP
>> >> > wheel file based on Python version in PATH.
>> >> >
>> >> > This pip package build setup is supported for python version 2.6 and
>> >> > above.
>> >> > The pip package can easily be built with a make statement.
>> >> >
>> >> > Makefile has following targets.
>> >> >   - install-required: To install packages required to build PIP
>> >> > package.
>> >> >   - all: To build PIP package
>> >> >   - install-pip: To install built PIP package
>> >> >   - clean: To clean the sources.
>> >>
>> >> Let's try to keep the Makefile target names specific to what we're
>> >> building, e.g.
>> >>
>> >> install-pip-requirements: To install packages required to build PIP
>> >> package (not sure that's the best name though)
>> >> install-pip: To install built PIP package
>> >> pip: To build the PIP package
>> >
>> >
>> > Done.
>> >>
>> >>
>> >> Eventually the "all"  target will likely build the runtime and maybe
>> >> other packages as well, so for now, it should probably look something
>> >> like:
>> >>
>> >> all: install-pip-requirements pip
>> >
>> >
>> > Done.
>> >>
>> >>
>> >> Similarly, "clean" will eventually do other things too, so we can
>> >> create "clean-pip", which can be called with other further clean
>> >> targets by "clean".
>> >
>> >
>> > Done.
>> >
>> >>
>> >>
>> >> > setup_pgadmin4.py file contains all the Meta information needed to
>> >> > build
>> >> > the
>> >> > PIP package.
>> >>
>> >> I think that should be moved to pkg/pip, where it can later live
>> >> alongside directories like win32, osx and rpm etc. Would setup_pip.py
>> >> be a better name as well?
>> >
>> >
>> > Done.
>> >
>> >>
>> >>
>> >> > The version for the pip package is picked up from web/config.py file.
>> >>
>> >> Urgh - don't try to parse that file from a Python script. It's a
>> >> Python script itself, so you can just import it - e.g.
>> >>
>> >> import config
>> >> version = config.APP_VERSION
>> >
>> >
>> > Done.
>> >
>> >>
>> >>
>> >> Thanks!
>> >>
>> >> --
>> >> Dave Page
>> >> Blog: http://pgsnake.blogspot.com
>> >> Twitter: @pgsnake
>> >>
>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >
>> >
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

On Mon, Mar 14, 2016 at 1:46 PM, Muhammad Aqeel
<muhammad.aqeel@enterprisedb.com> wrote:
> Hi Dave,
>
> I tested the build after your commit. There were minor bugs in your changes
> which prevented the packaging of pgadmin4 sources.
>
> - It needed a space after * in recursive include statement of manifest.in
> - \n is not interpreted as new line.
>
> I fixed both issues in attached patch and tested the build. It is working
> fine now and pgadmin4.egg-info/SOURCES.txt showing no .pyc file and
> pgadmin4.db file in it.

Weird - as discussed, I tested it thoroughly before committing and it
works fine.

Testing with this patch though, I see the original problem (and also,
so warnings - can you check them please)?

(pgadmin4)piranha:pgAdmin4 dpage$ git reset --hard
HEAD is now at dc64160 Add a top level Makefile with a target and
appropriate Python stuff to build a PIP package of the web code.
(pgadmin4)piranha:pgAdmin4 dpage$ make pip 1> /dev/null
/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/setuptools/dist.py:285:
UserWarning: Normalizing '1.0.0-dev' to '1.0.0.dev0'
  normalized_version,
package init file 'pgadmin4/__init__.py' not found (or not a regular file)
warning: manifest_maker: standard file 'setup_pip.py' not found

warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

(pgadmin4)piranha:pgAdmin4 dpage$ grep pgadmin4.db
pgadmin4.egg-info/SOURCES.txt |wc -l
       0
(pgadmin4)piranha:pgAdmin4 dpage$ grep pyc pgadmin4.egg-info/SOURCES.txt |wc -l
       0
(pgadmin4)piranha:pgAdmin4 dpage$ git apply ~/Downloads/manifest_bug_fix.patch
(pgadmin4)piranha:pgAdmin4 dpage$ make pip 1> /dev/null
/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/setuptools/dist.py:285:
UserWarning: Normalizing '1.0.0-dev' to '1.0.0.dev0'
  normalized_version,
package init file 'pgadmin4/__init__.py' not found (or not a regular file)
warning: manifest_maker: standard file 'setup_pip.py' not found

warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

(pgadmin4)piranha:pgAdmin4 dpage$ grep pgadmin4.db
pgadmin4.egg-info/SOURCES.txt |wc -l
       1
(pgadmin4)piranha:pgAdmin4 dpage$ grep pyc pgadmin4.egg-info/SOURCES.txt |wc -l
      84

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

On Tue, Mar 15, 2016 at 6:23 AM, Muhammad Aqeel
<muhammad.aqeel@enterprisedb.com> wrote:
> Hi Dave,
>
> On Mon, Mar 14, 2016 at 7:06 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi
>>
>> On Mon, Mar 14, 2016 at 1:46 PM, Muhammad Aqeel
>> <muhammad.aqeel@enterprisedb.com> wrote:
>> > Hi Dave,
>> >
>> > I tested the build after your commit. There were minor bugs in your
>> > changes
>> > which prevented the packaging of pgadmin4 sources.
>> >
>> > - It needed a space after * in recursive include statement of
>> > manifest.in
>> > - \n is not interpreted as new line.
>> >
>> > I fixed both issues in attached patch and tested the build. It is
>> > working
>> > fine now and pgadmin4.egg-info/SOURCES.txt showing no .pyc file and
>> > pgadmin4.db file in it.
>>
>> Weird - as discussed, I tested it thoroughly before committing and it
>> works fine.
>
>
> PFA the new patch which contains following changes.
>
> - It just replaces the @echo to @printf in your changed line to fix the
> issue. I tested it on both Linux and OSX. It is working fine.

Looks good to me - committed.

> - In your commit you had changed 'PGADMIN_DIST = dist' to 'PGADMIN_DIST =
> build'. dist directory is generated by pip setuptools to have generated
> wheel file so it is needed. I made it again 'PGADMIN_DIST = dist'. If you
> don't want to remove it in clean-pip target then you can remove the rm
> command from clean-pip target.

Urgh. It might be nice if we can put all those directories in one
place, like pip-temp/ or something more sensible. Either way, it's no
big issue.

> - In case user runs make twice without cleaning the source it was copying
> the web directory to existing pgadmin4 source directory. I fixed this issue
> too.

OK, thanks.

>> Testing with this patch though, I see the original problem (and also,
>> so warnings - can you check them please)?
>>
>> (pgadmin4)piranha:pgAdmin4 dpage$ git reset --hard
>> HEAD is now at dc64160 Add a top level Makefile with a target and
>> appropriate Python stuff to build a PIP package of the web code.
>> (pgadmin4)piranha:pgAdmin4 dpage$ make pip 1> /dev/null
>>
>> /Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/setuptools/dist.py:285:
>> UserWarning: Normalizing '1.0.0-dev' to '1.0.0.dev0'
>>   normalized_version,
>
>
> It is an issue with PEP440 which interprets version as integer and if sees
> any string in it, it tries to normalise it. Here is the link which explains
> it.
>
> https://bitbucket.org/pypa/setuptools/issues/302/setuptools-8-strips-leading-zeros-in
>
> It is coming with Python 2.6 only. With Python 2.7 and above it is not
> there.

Weird - I have 2.7...

>> package init file 'pgadmin4/__init__.py' not found (or not a regular file)
>
>
> This is because in web directory we don't have __init__.py file.
>
>>
>> warning: manifest_maker: standard file 'setup_pip.py' not found
>
>
> setuptools by default look for setup_pip.py file in current directory. Since
> we placed this file in pkg/pip (according to your suggestion) this warning
> is coming.
>
>>
>>
>> warning: build_py: byte-compiling is disabled, skipping.
>>
>> warning: install_lib: byte-compiling is disabled, skipping.
>
>
> I have not faced this issue but following link suggests a solution for it.
>
> http://stackoverflow.com/questions/7194299/cannot-create-new-virtualenv-in-lion
>
> It says "Setting export VIRTUALENV_USE_DISTRIBUTE=1 fixed the problem."

Oh - that's probably because I have "export PYTHONDONTWRITEBYTECODE=1"
in my profile.

No worries - thanks!


>> (pgadmin4)piranha:pgAdmin4 dpage$ grep pgadmin4.db
>> pgadmin4.egg-info/SOURCES.txt |wc -l
>>        0
>> (pgadmin4)piranha:pgAdmin4 dpage$ grep pyc pgadmin4.egg-info/SOURCES.txt
>> |wc -l
>>        0
>> (pgadmin4)piranha:pgAdmin4 dpage$ git apply
>> ~/Downloads/manifest_bug_fix.patch
>> (pgadmin4)piranha:pgAdmin4 dpage$ make pip 1> /dev/null
>>
>> /Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/setuptools/dist.py:285:
>> UserWarning: Normalizing '1.0.0-dev' to '1.0.0.dev0'
>>   normalized_version,
>> package init file 'pgadmin4/__init__.py' not found (or not a regular file)
>> warning: manifest_maker: standard file 'setup_pip.py' not found
>>
>> warning: build_py: byte-compiling is disabled, skipping.
>>
>> warning: install_lib: byte-compiling is disabled, skipping.
>>
>> (pgadmin4)piranha:pgAdmin4 dpage$ grep pgadmin4.db
>> pgadmin4.egg-info/SOURCES.txt |wc -l
>>        1
>> (pgadmin4)piranha:pgAdmin4 dpage$ grep pyc pgadmin4.egg-info/SOURCES.txt
>> |wc -l
>>       84
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company