Thread: Unified server/desktop config

Hi Dave,

Partial patch for feature test got included in you patch :)

On Thu, Jul 20, 2017 at 9:33 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Oops sorry my bad, I did not removed it.

--

Regards,

Murtuza Zabuawala
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Jul 20, 2017 at 10:22 PM, Murtuza Zabuawala <murtuza.zabuawala@enterprisedb.com> wrote:

Hi Dave,

Partial patch for feature test got included in you patch :)

On Thu, Jul 20, 2017 at 9:33 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Aug 7, 2017 at 4:11 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 3:59 PM, Dave Page <dpage@pgadmin.org> wrote:

Anyone?

Surinder - please give this one priority.

Sure

​.​

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

On Thu, Jul 20, 2017 at 5:03 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Please update the patch :-)

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK:http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 9 Aug 2017, at 05:53, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

I noticed that test cases don’t run and I got an error:

(pgAdmin_27)Laptop195:regression surinder$ python runtests.py --pkg feature_tests
<module '__builtin__' (built-in)>
Traceback (most recent call last): File "runtests.py", line 45, in <module>   import config File "/Users/surinder/Documents/Projects/pgadmin4/web/config.py", line 121, in <module>   if builtins.SERVER_MODE is None:
AttributeError: 'module' object has no attribute'
SERVER_MODE
'

I think this is because we are setting​ ​first​ ​builtins.SERVER_MODE​ in​ ​pgAdmin4.py
but in test cases pgAdmin4.py is not being called.​

On Tue, Aug 8, 2017 at 4:03 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 11:27 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Tue, Aug 8, 2017 at 1:47 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 7:18 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

On Ubuntu-14.04, I got error Application Server couldn't be contacted:

Steps performed:

• I have already installed pgAdmin4-1.4 which come with PostgreSQL-9.6 installer.
  then I run root@ubuntu:/opt/PostgreSQL/9.6/pgAdmin 4/bin# ./pgAdmin4./.
• Now took latest git pull from HEAD
• Apply unified_config.diff patch.
• Then compiled pgAdmin4 in runtime and then run ./pgAdmin.
• Got error Application Server couldn't be contacted.

But when I ran ./pgAdmin4 for the second time. pgAdmin4 runs without any issue.
I didn’t get any error on the terminal and log file.

​ I couldn't find why it gives this error.​

I know Fahar has run into this with existing releases on Ubuntu. If you enable debugging, can you see any clues? I assume it's going round the retry loop before aborting? 

Another issue related to Alembic:

If I am running pgAdmin4 already installed on my machine, then I upgrade pgAdmin4 using Python wheel:

(test_p27) surinder@ubuntu:~/virtualenvs/test_p27$ python ~/virtualenvs/test_p27/lib/python2.7/site-packages/pgadmin4/pgAdmin4.py

I am getting error:

alembic.util.exc.CommandError: Can't locate revision identified by 'd85a62333272'

To fix this, I have to delete existing pgadmin4.db file. I don’t know if it is a valid case or should I log an RM if it is?

That's an annoying side-effect of the move to Alembic. The previous DB code would quite happily (and intentionally) run with a newer version of the database, however, the new code does not. You'll see this issue whenever you've run a newer version of pgAdmin and then go back to an older version that uses an older schema version.

It would be nice to change this so it doesn't complain - but we'd have to be cautious to only break compatibility on major releases.

 

Apart for this, I didn’t see any functionality break. It works!!

​

I liked the approach to set SEVER_MODE in runtime using built-ins.

Thanks. Do you think it warrants a 2.0 version number, given the potential for breaking existing installations (I do, but would like other feedback)? If we do that - and thus allow a parallel installation of 1.x and 2.x), how would we resolve the Alembic issue you noted such that both versions could be run against the same DB?

To resolve this issue while upgrading to newer version of pgAdmin4, we can add a parameter version_table: 'alembic_version-1.6 in web/migrations/env.py of upcoming pgAdmin4 source code(reference link)
It will create a new version table in pgadmin4.db database and it will instruct the Alembic to run migrations for the version stored in table(alembic_version-1.6) when new pgAdmin4 will be installed.

I tried to check if it works or not. But somehow I couldn’t install pgAdmin4-1.5 from wheel.

Alternatively I will try to generate new schema using in pgAdmin4 source code(Git HEAD). It will generate a new migration version and then i will run pgAdmin4-1.6 from wheel having different migration number. It will create pgadmin4.db for 1.5.

Then

 I will run pgAdmin4-1.6 from source code, so migration will take place and hopefully

​the ​

sa

me

​ ​

pgadmin4.db will work for newer pgAdmin4 version.

Reference link where the same issue has been discussed.

Sounds good - please investigate further.

 I spent some time, understands how Alembic works and I thought it is easy using the reference link but not, It needs more R&D. I will look how can we fix it.

Thanks!

 

Thanks,
Surinder

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 7:01 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, Aug 7, 2017 at 1:43 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

The patch seems to work in Runtime mode, but fails in Server mode with error:

(pgAdmin_27)Laptop195:pgadmin4 surinder$ python web/pgAdmin4.py
Traceback (most recent call last): File "web/pgAdmin4.py", line 55, in <module>   exec(open(file_quote(setupfile), 'r').read()) File "<string>", line 35, in <module> File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 23, in create_app_data_directory   _create_directory_if_not_exists(os.path.dirname(config.SQLITE_PATH)) File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 15, in _create_directory_if_not_exists   os.mkdir(_path)
OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'
(pgAdmin_27)Laptop195:pgadmin4 surinder$

This is because the directory /var/lib/ has root only access and I am running pgAdmin4 with the non-root user.

Also pgadmin directory is not created.

(pgAdmin_35)Laptop195:pgadmin4 surinder$ ls /var/lib/pgadmin
ls: /var/lib/pgadmin: No such file or directory

I got same error with MacOSX and Ubuntu-14.04 machines irrespective of Python version.

Meanwhile, I am testing patch with other test cases.

That's fully expected. In the case of Linux, the packages will be responsible for creating those directories with the appropriate ownership. In other cases, the user would.

​ok, I got it.​

There's not really much we can do about that - and it's exactly what would happen if you try to run many other packages yourself when standard *nix paths are used. 

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 4:33 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 4:11 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 3:59 PM, Dave Page <dpage@pgadmin.org> wrote:

Anyone?

Surinder - please give this one priority.

Sure

​.​

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

On Thu, Jul 20, 2017 at 5:03 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

​

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi

This patch includes the fix where this patch breaks when feature tests are run.

Set builtins.SERVER_MODE if it is present in globals()['SERVER_MODE'] else set to None.

Please find updated patch.Thanks,
Surinder

​

On Wed, Aug 9, 2017 at 11:57 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Sure, I will update.

On Wed, Aug 9, 2017 at 11:17 AM, Dave Page <dpage@pgadmin.org> wrote:

Please update the patch :-)

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK:http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 9 Aug 2017, at 05:53, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

I noticed that test cases don’t run and I got an error:

(pgAdmin_27)Laptop195:regression surinder$ python runtests.py --pkg feature_tests
<module '__builtin__' (built-in)>
Traceback (most recent call last): File "runtests.py", line 45, in <module>   import config File "/Users/surinder/Documents/Projects/pgadmin4/web/config.py", line 121, in <module>   if builtins.SERVER_MODE is None:
AttributeError: 'module' object has no attribute'
SERVER_MODE
'

I think this is because we are setting​ ​first​ ​builtins.SERVER_MODE​ in​ ​pgAdmin4.py
but in test cases pgAdmin4.py is not being called.​

On Tue, Aug 8, 2017 at 4:03 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 11:27 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Tue, Aug 8, 2017 at 1:47 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 7:18 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

On Ubuntu-14.04, I got error Application Server couldn't be contacted:

Steps performed:

• I have already installed pgAdmin4-1.4 which come with PostgreSQL-9.6 installer.
  then I run root@ubuntu:/opt/PostgreSQL/9.6/pgAdmin 4/bin# ./pgAdmin4./.
• Now took latest git pull from HEAD
• Apply unified_config.diff patch.
• Then compiled pgAdmin4 in runtime and then run ./pgAdmin.
• Got error Application Server couldn't be contacted.

But when I ran ./pgAdmin4 for the second time. pgAdmin4 runs without any issue.
I didn’t get any error on the terminal and log file.

​ I couldn't find why it gives this error.​

I know Fahar has run into this with existing releases on Ubuntu. If you enable debugging, can you see any clues? I assume it's going round the retry loop before aborting? 

Another issue related to Alembic:

If I am running pgAdmin4 already installed on my machine, then I upgrade pgAdmin4 using Python wheel:

(test_p27) surinder@ubuntu:~/virtualenvs/test_p27$ python ~/virtualenvs/test_p27/lib/python2.7/site-packages/pgadmin4/pgAdmin4.py

I am getting error:

alembic.util.exc.CommandError: Can't locate revision identified by 'd85a62333272'

To fix this, I have to delete existing pgadmin4.db file. I don’t know if it is a valid case or should I log an RM if it is?

That's an annoying side-effect of the move to Alembic. The previous DB code would quite happily (and intentionally) run with a newer version of the database, however, the new code does not. You'll see this issue whenever you've run a newer version of pgAdmin and then go back to an older version that uses an older schema version.

It would be nice to change this so it doesn't complain - but we'd have to be cautious to only break compatibility on major releases.

 

Apart for this, I didn’t see any functionality break. It works!!

​

I liked the approach to set SEVER_MODE in runtime using built-ins.

Thanks. Do you think it warrants a 2.0 version number, given the potential for breaking existing installations (I do, but would like other feedback)? If we do that - and thus allow a parallel installation of 1.x and 2.x), how would we resolve the Alembic issue you noted such that both versions could be run against the same DB?

To resolve this issue while upgrading to newer version of pgAdmin4, we can add a parameter version_table: 'alembic_version-1.6 in web/migrations/env.py of upcoming pgAdmin4 source code(reference link)
It will create a new version table in pgadmin4.db database and it will instruct the Alembic to run migrations for the version stored in table(alembic_version-1.6) when new pgAdmin4 will be installed.

I tried to check if it works or not. But somehow I couldn’t install pgAdmin4-1.5 from wheel.

Alternatively I will try to generate new schema using in pgAdmin4 source code(Git HEAD). It will generate a new migration version and then i will run pgAdmin4-1.6 from wheel having different migration number. It will create pgadmin4.db for 1.5.

Then

 I will run pgAdmin4-1.6 from source code, so migration will take place and hopefully

​the ​

sa

me

​ ​

pgadmin4.db will work for newer pgAdmin4 version.

Reference link where the same issue has been discussed.

Sounds good - please investigate further.

 I spent some time, understands how Alembic works and I thought it is easy using the reference link but not, It needs more R&D. I will look how can we fix it.

Thanks!

 

Thanks,
Surinder

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 7:01 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, Aug 7, 2017 at 1:43 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

The patch seems to work in Runtime mode, but fails in Server mode with error:

(pgAdmin_27)Laptop195:pgadmin4 surinder$ python web/pgAdmin4.py
Traceback (most recent call last): File "web/pgAdmin4.py", line 55, in <module>   exec(open(file_quote(setupfile), 'r').read()) File "<string>", line 35, in <module> File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 23, in create_app_data_directory   _create_directory_if_not_exists(os.path.dirname(config.SQLITE_PATH)) File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 15, in _create_directory_if_not_exists   os.mkdir(_path)
OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'
(pgAdmin_27)Laptop195:pgadmin4 surinder$

This is because the directory /var/lib/ has root only access and I am running pgAdmin4 with the non-root user.

Also pgadmin directory is not created.

(pgAdmin_35)Laptop195:pgadmin4 surinder$ ls /var/lib/pgadmin
ls: /var/lib/pgadmin: No such file or directory

I got same error with MacOSX and Ubuntu-14.04 machines irrespective of Python version.

Meanwhile, I am testing patch with other test cases.

That's fully expected. In the case of Linux, the packages will be responsible for creating those directories with the appropriate ownership. In other cases, the user would.

​ok, I got it.​

There's not really much we can do about that - and it's exactly what would happen if you try to run many other packages yourself when standard *nix paths are used. 

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 4:33 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 4:11 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 3:59 PM, Dave Page <dpage@pgadmin.org> wrote:

Anyone?

Surinder - please give this one priority.

Sure

​.​

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

On Thu, Jul 20, 2017 at 5:03 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

​

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Thanks. Out of interest, what changes were required?

All, this reinforces to me that this should be a v2.0 change. Other thoughts?

On Mon, Aug 14, 2017 at 4:59 AM, Bing Xu <bxu@pivotal.io> wrote:

Hi hackers,

We've reviewed this patch and it need us changing our own config_local.py file. It's ok and we'll do the change in our configuration.

Thanks,

Bing & Violet

On Wed, Aug 9, 2017 at 8:49 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

This patch includes the fix where this patch breaks when feature tests are run.

Set builtins.SERVER_MODE if it is present in globals()['SERVER_MODE'] else set to None.

Please find updated patch.Thanks,
Surinder

​

On Wed, Aug 9, 2017 at 11:57 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Sure, I will update.

On Wed, Aug 9, 2017 at 11:17 AM, Dave Page <dpage@pgadmin.org> wrote:

Please update the patch :-)

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK:http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 9 Aug 2017, at 05:53, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

I noticed that test cases don’t run and I got an error:

(pgAdmin_27)Laptop195:regression surinder$ python runtests.py --pkg feature_tests
<module '__builtin__' (built-in)>
Traceback (most recent call last): File "runtests.py", line 45, in <module>   import config File "/Users/surinder/Documents/Projects/pgadmin4/web/config.py", line 121, in <module>   if builtins.SERVER_MODE is None:
AttributeError: 'module' object has no attribute'
SERVER_MODE
'

I think this is because we are setting​ ​first​ ​builtins.SERVER_MODE​ in​ ​pgAdmin4.py
but in test cases pgAdmin4.py is not being called.​

On Tue, Aug 8, 2017 at 4:03 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 11:27 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Tue, Aug 8, 2017 at 1:47 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 7:18 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

On Ubuntu-14.04, I got error Application Server couldn't be contacted:

Steps performed:

• I have already installed pgAdmin4-1.4 which come with PostgreSQL-9.6 installer.
  then I run root@ubuntu:/opt/PostgreSQL/9.6/pgAdmin 4/bin# ./pgAdmin4./.
• Now took latest git pull from HEAD
• Apply unified_config.diff patch.
• Then compiled pgAdmin4 in runtime and then run ./pgAdmin.
• Got error Application Server couldn't be contacted.

But when I ran ./pgAdmin4 for the second time. pgAdmin4 runs without any issue.
I didn’t get any error on the terminal and log file.

​ I couldn't find why it gives this error.​

I know Fahar has run into this with existing releases on Ubuntu. If you enable debugging, can you see any clues? I assume it's going round the retry loop before aborting? 

Another issue related to Alembic:

If I am running pgAdmin4 already installed on my machine, then I upgrade pgAdmin4 using Python wheel:

(test_p27) surinder@ubuntu:~/virtualenvs/test_p27$ python ~/virtualenvs/test_p27/lib/python2.7/site-packages/pgadmin4/pgAdmin4.py

I am getting error:

alembic.util.exc.CommandError: Can't locate revision identified by 'd85a62333272'

To fix this, I have to delete existing pgadmin4.db file. I don’t know if it is a valid case or should I log an RM if it is?

That's an annoying side-effect of the move to Alembic. The previous DB code would quite happily (and intentionally) run with a newer version of the database, however, the new code does not. You'll see this issue whenever you've run a newer version of pgAdmin and then go back to an older version that uses an older schema version.

It would be nice to change this so it doesn't complain - but we'd have to be cautious to only break compatibility on major releases.

 

Apart for this, I didn’t see any functionality break. It works!!

​

I liked the approach to set SEVER_MODE in runtime using built-ins.

Thanks. Do you think it warrants a 2.0 version number, given the potential for breaking existing installations (I do, but would like other feedback)? If we do that - and thus allow a parallel installation of 1.x and 2.x), how would we resolve the Alembic issue you noted such that both versions could be run against the same DB?

To resolve this issue while upgrading to newer version of pgAdmin4, we can add a parameter version_table: 'alembic_version-1.6 in web/migrations/env.py of upcoming pgAdmin4 source code(reference link)
It will create a new version table in pgadmin4.db database and it will instruct the Alembic to run migrations for the version stored in table(alembic_version-1.6) when new pgAdmin4 will be installed.

I tried to check if it works or not. But somehow I couldn’t install pgAdmin4-1.5 from wheel.

Alternatively I will try to generate new schema using in pgAdmin4 source code(Git HEAD). It will generate a new migration version and then i will run pgAdmin4-1.6 from wheel having different migration number. It will create pgadmin4.db for 1.5.

Then

 I will run pgAdmin4-1.6 from source code, so migration will take place and hopefully

​the ​

sa

me

​ ​

pgadmin4.db will work for newer pgAdmin4 version.

Reference link where the same issue has been discussed.

Sounds good - please investigate further.

 I spent some time, understands how Alembic works and I thought it is easy using the reference link but not, It needs more R&D. I will look how can we fix it.

Thanks!

 

Thanks,
Surinder

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 7:01 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, Aug 7, 2017 at 1:43 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

The patch seems to work in Runtime mode, but fails in Server mode with error:

(pgAdmin_27)Laptop195:pgadmin4 surinder$ python web/pgAdmin4.py
Traceback (most recent call last): File "web/pgAdmin4.py", line 55, in <module>   exec(open(file_quote(setupfile), 'r').read()) File "<string>", line 35, in <module> File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 23, in create_app_data_directory   _create_directory_if_not_exists(os.path.dirname(config.SQLITE_PATH)) File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 15, in _create_directory_if_not_exists   os.mkdir(_path)
OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'
(pgAdmin_27)Laptop195:pgadmin4 surinder$

This is because the directory /var/lib/ has root only access and I am running pgAdmin4 with the non-root user.

Also pgadmin directory is not created.

(pgAdmin_35)Laptop195:pgadmin4 surinder$ ls /var/lib/pgadmin
ls: /var/lib/pgadmin: No such file or directory

I got same error with MacOSX and Ubuntu-14.04 machines irrespective of Python version.

Meanwhile, I am testing patch with other test cases.

That's fully expected. In the case of Linux, the packages will be responsible for creating those directories with the appropriate ownership. In other cases, the user would.

​ok, I got it.​

There's not really much we can do about that - and it's exactly what would happen if you try to run many other packages yourself when standard *nix paths are used. 

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 4:33 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 4:11 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 3:59 PM, Dave Page <dpage@pgadmin.org> wrote:

Anyone?

Surinder - please give this one priority.

Sure

​.​

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

On Thu, Jul 20, 2017 at 5:03 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

​

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi, Dave

Our previous config_local.py is:

===================================================================

from config import *
	DEBUG = True
	# Enable the test module
	MODULE_BLACKLIST.remove('test')
	# Log
	CONSOLE_LOG_LEVEL = DEBUG
	FILE_LOG_LEVEL = DEBUG
	DEFAULT_SERVER = '127.0.0.1'
	UPGRADE_CHECK_ENABLED = True
	SERVER_MODE = False
	# Use a different config DB for each server mode.
	if SERVER_MODE == False:
	SQLITE_PATH = os.path.join(
	DATA_DIR,
	'pgadmin4-desktop.db'
	)
	else:
	SQLITE_PATH = os.path.join(
	DATA_DIR,
	)

==============================================================

Now we need config 

DATA_DIR, LOG_FILE and all the configurations related to DATA_DIR

Thanks,

 Violet & Bing 

On Thu, Aug 17, 2017 at 4:23 PM, Dave Page <dpage@pgadmin.org> wrote:

Thanks. Out of interest, what changes were required?

All, this reinforces to me that this should be a v2.0 change. Other thoughts?

On Mon, Aug 14, 2017 at 4:59 AM, Bing Xu <bxu@pivotal.io> wrote:

Hi hackers,

We've reviewed this patch and it need us changing our own config_local.py file. It's ok and we'll do the change in our configuration.

Thanks,

Bing & Violet

On Wed, Aug 9, 2017 at 8:49 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

This patch includes the fix where this patch breaks when feature tests are run.

Set builtins.SERVER_MODE if it is present in globals()['SERVER_MODE'] else set to None.

Please find updated patch.Thanks,
Surinder

​

On Wed, Aug 9, 2017 at 11:57 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Sure, I will update.

On Wed, Aug 9, 2017 at 11:17 AM, Dave Page <dpage@pgadmin.org> wrote:

Please update the patch :-)

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK:http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 9 Aug 2017, at 05:53, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

I noticed that test cases don’t run and I got an error:

(pgAdmin_27)Laptop195:regression surinder$ python runtests.py --pkg feature_tests
<module '__builtin__' (built-in)>
Traceback (most recent call last): File "runtests.py", line 45, in <module>   import config File "/Users/surinder/Documents/Projects/pgadmin4/web/config.py", line 121, in <module>   if builtins.SERVER_MODE is None:
AttributeError: 'module' object has no attribute'
SERVER_MODE
'

I think this is because we are setting​ ​first​ ​builtins.SERVER_MODE​ in​ ​pgAdmin4.py
but in test cases pgAdmin4.py is not being called.​

On Tue, Aug 8, 2017 at 4:03 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 11:27 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Tue, Aug 8, 2017 at 1:47 PM, Dave Page <dpage@pgadmin.org> wrote:

On Tue, Aug 8, 2017 at 7:18 AM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi

On Ubuntu-14.04, I got error Application Server couldn't be contacted:

Steps performed:

• I have already installed pgAdmin4-1.4 which come with PostgreSQL-9.6 installer.
  then I run root@ubuntu:/opt/PostgreSQL/9.6/pgAdmin 4/bin# ./pgAdmin4./.
• Now took latest git pull from HEAD
• Apply unified_config.diff patch.
• Then compiled pgAdmin4 in runtime and then run ./pgAdmin.
• Got error Application Server couldn't be contacted.

But when I ran ./pgAdmin4 for the second time. pgAdmin4 runs without any issue.
I didn’t get any error on the terminal and log file.

​ I couldn't find why it gives this error.​

I know Fahar has run into this with existing releases on Ubuntu. If you enable debugging, can you see any clues? I assume it's going round the retry loop before aborting? 

Another issue related to Alembic:

If I am running pgAdmin4 already installed on my machine, then I upgrade pgAdmin4 using Python wheel:

(test_p27) surinder@ubuntu:~/virtualenvs/test_p27$ python ~/virtualenvs/test_p27/lib/python2.7/site-packages/pgadmin4/pgAdmin4.py

I am getting error:

alembic.util.exc.CommandError: Can't locate revision identified by 'd85a62333272'

To fix this, I have to delete existing pgadmin4.db file. I don’t know if it is a valid case or should I log an RM if it is?

That's an annoying side-effect of the move to Alembic. The previous DB code would quite happily (and intentionally) run with a newer version of the database, however, the new code does not. You'll see this issue whenever you've run a newer version of pgAdmin and then go back to an older version that uses an older schema version.

It would be nice to change this so it doesn't complain - but we'd have to be cautious to only break compatibility on major releases.

 

Apart for this, I didn’t see any functionality break. It works!!

​

I liked the approach to set SEVER_MODE in runtime using built-ins.

Thanks. Do you think it warrants a 2.0 version number, given the potential for breaking existing installations (I do, but would like other feedback)? If we do that - and thus allow a parallel installation of 1.x and 2.x), how would we resolve the Alembic issue you noted such that both versions could be run against the same DB?

To resolve this issue while upgrading to newer version of pgAdmin4, we can add a parameter version_table: 'alembic_version-1.6 in web/migrations/env.py of upcoming pgAdmin4 source code(reference link)
It will create a new version table in pgadmin4.db database and it will instruct the Alembic to run migrations for the version stored in table(alembic_version-1.6) when new pgAdmin4 will be installed.

I tried to check if it works or not. But somehow I couldn’t install pgAdmin4-1.5 from wheel.

Alternatively I will try to generate new schema using in pgAdmin4 source code(Git HEAD). It will generate a new migration version and then i will run pgAdmin4-1.6 from wheel having different migration number. It will create pgadmin4.db for 1.5.

Then

 I will run pgAdmin4-1.6 from source code, so migration will take place and hopefully

​the ​

sa

me

​ ​

pgadmin4.db will work for newer pgAdmin4 version.

Reference link where the same issue has been discussed.

Sounds good - please investigate further.

 I spent some time, understands how Alembic works and I thought it is easy using the reference link but not, It needs more R&D. I will look how can we fix it.

Thanks!

 

Thanks,
Surinder

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 7:01 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, Aug 7, 2017 at 1:43 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

Hi,

The patch seems to work in Runtime mode, but fails in Server mode with error:

(pgAdmin_27)Laptop195:pgadmin4 surinder$ python web/pgAdmin4.py
Traceback (most recent call last): File "web/pgAdmin4.py", line 55, in <module>   exec(open(file_quote(setupfile), 'r').read()) File "<string>", line 35, in <module> File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 23, in create_app_data_directory   _create_directory_if_not_exists(os.path.dirname(config.SQLITE_PATH)) File "/Users/surinder/Documents/Projects/pgadmin4/web/pgadmin/setup/data_directory.py", line 15, in _create_directory_if_not_exists   os.mkdir(_path)
OSError: [Errno 13] Permission denied: '/var/lib/pgadmin'
(pgAdmin_27)Laptop195:pgadmin4 surinder$

This is because the directory /var/lib/ has root only access and I am running pgAdmin4 with the non-root user.

Also pgadmin directory is not created.

(pgAdmin_35)Laptop195:pgadmin4 surinder$ ls /var/lib/pgadmin
ls: /var/lib/pgadmin: No such file or directory

I got same error with MacOSX and Ubuntu-14.04 machines irrespective of Python version.

Meanwhile, I am testing patch with other test cases.

That's fully expected. In the case of Linux, the packages will be responsible for creating those directories with the appropriate ownership. In other cases, the user would.

​ok, I got it.​

There's not really much we can do about that - and it's exactly what would happen if you try to run many other packages yourself when standard *nix paths are used. 

 

Thanks,
Surinder

​

On Mon, Aug 7, 2017 at 4:33 PM, Surinder Kumar <surinder.kumar@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 4:11 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Mon, Aug 7, 2017 at 3:59 PM, Dave Page <dpage@pgadmin.org> wrote:

Anyone?

Surinder - please give this one priority.

Sure

​.​

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

On Thu, Jul 20, 2017 at 5:03 PM, Dave Page <dpage@pgadmin.org> wrote:

All,

Attached is a patch that aims to allow us to have a standardised config that will work out of the box for both web and desktop modes. It does this by doing two things:

1) The runtime sets SERVER_MODE in the Python environment before starting the app. If this value is set, then it overrides the default value of SERVER_MODE in the config.

2) The config file then offers default values for the various file locations for both server and desktop mode, setting them appropriately based on the derived SERVER_MODE value.

The only downsides I can see from this are:

- You cannot run in server mode in the runtime without manually reconfiguring SERVER_MODE and likely a bunch of paths in config_local.py

- If you want to override SERVER_MODE, you'll probably also need to redefine the various paths in config_local.py.

I don't see either being something 99.9% of users would need though.

Can anyone see if the patch breaks anything, or if I missed any side effects?

Is it likely to break things during upgrades? I suspect so... so maybe this should prompt v2.0?

I'd appreciate multiple reviews of this, as it could break things. Note that I haven't yet updated the docs.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

​

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company