Thread: [Patch] Fix Unicode in errmsg
Hi All, Just a small patch to fix server addition/connection errors when using non-ascii languages (pt_BR in my case). I haven't delved enough into the codebase to be sure, but there might be more occurrences of errmsg that need decoding from ASCII to UTF-8. Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be maintained until 2020? Python 3 support was added in Flask 0.10 and, as mentioned in the Flask Docs, "Most libraries and Flask extensions have been ported by now and using Flask with Python 3 is generally a smooth ride.". Thanks for this amazing piece of software. -- Magnun Leno
Attachment
Thanks Magnun. Ashesh, can you review this please, and check if similar changes are needed elsewhere? Thanks. On Wed, Jul 6, 2016 at 7:55 PM, Magnun Leno (Pessoal) <magnun.leno@gmail.com> wrote: > Hi All, > > Just a small patch to fix server addition/connection errors when using > non-ascii languages (pt_BR in my case). I haven't delved enough into the > codebase to be sure, but there might be more occurrences of errmsg that need > decoding from ASCII to UTF-8. > > Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be > maintained until 2020? Python 3 support was added in Flask 0.10 and, as > mentioned in the Flask Docs, "Most libraries and Flask extensions have been > ported by now and using Flask with Python 3 is generally a smooth ride.". > > Thanks for this amazing piece of software. > > -- > Magnun Leno > > > -- > Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgadmin-hackers > -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Attachment
Sure - I will do.
On Thu, Jul 7, 2016 at 2:31 PM, Dave Page <dpage@pgadmin.org> wrote:
Thanks Magnun.
Ashesh, can you review this please, and check if similar changes are
needed elsewhere?
Thanks.> --
On Wed, Jul 6, 2016 at 7:55 PM, Magnun Leno (Pessoal)
<magnun.leno@gmail.com> wrote:
> Hi All,
>
> Just a small patch to fix server addition/connection errors when using
> non-ascii languages (pt_BR in my case). I haven't delved enough into the
> codebase to be sure, but there might be more occurrences of errmsg that need
> decoding from ASCII to UTF-8.
>
> Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be
> maintained until 2020? Python 3 support was added in Flask 0.10 and, as
> mentioned in the Flask Docs, "Most libraries and Flask extensions have been
> ported by now and using Flask with Python 3 is generally a smooth ride.".
>
> Thanks for this amazing piece of software.
>
> --
> Magnun Leno
>
>
> Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgadmin-hackers
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Just found another decoding error, this time during server group creation. Path attached.
Thanks again.
---
Magnun Leno
Em 07-07-2016 08:39, Ashesh Vashi escreveu:
Sure - I will do.On Thu, Jul 7, 2016 at 2:31 PM, Dave Page <dpage@pgadmin.org> wrote:Thanks Magnun.
Ashesh, can you review this please, and check if similar changes are
needed elsewhere?
Thanks.> --
On Wed, Jul 6, 2016 at 7:55 PM, Magnun Leno (Pessoal)
<magnun.leno@gmail.com> wrote:
> Hi All,
>
> Just a small patch to fix server addition/connection errors when using
> non-ascii languages (pt_BR in my case). I haven't delved enough into the
> codebase to be sure, but there might be more occurrences of errmsg that need
> decoding from ASCII to UTF-8.
>
> Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be
> maintained until 2020? Python 3 support was added in Flask 0.10 and, as
> mentioned in the Flask Docs, "Most libraries and Flask extensions have been
> ported by now and using Flask with Python 3 is generally a smooth ride.".
>
> Thanks for this amazing piece of software.
>
> --
> Magnun Leno
>
>
> Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgadmin-hackers
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachment
Ashesh, please review/commit this one too.
Thanks.
On Fri, Jul 8, 2016 at 5:07 PM, Magnun Leno (Pessoal) <magnun.leno@gmail.com> wrote:
Just found another decoding error, this time during server group creation. Path attached.
Thanks again.
---
Magnun Leno
Em 07-07-2016 08:39, Ashesh Vashi escreveu:Sure - I will do.On Thu, Jul 7, 2016 at 2:31 PM, Dave Page <dpage@pgadmin.org> wrote:Thanks Magnun.
Ashesh, can you review this please, and check if similar changes are
needed elsewhere?
Thanks.> --
On Wed, Jul 6, 2016 at 7:55 PM, Magnun Leno (Pessoal)
<magnun.leno@gmail.com> wrote:
> Hi All,
>
> Just a small patch to fix server addition/connection errors when using
> non-ascii languages (pt_BR in my case). I haven't delved enough into the
> codebase to be sure, but there might be more occurrences of errmsg that need
> decoding from ASCII to UTF-8.
>
> Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be
> maintained until 2020? Python 3 support was added in Flask 0.10 and, as
> mentioned in the Flask Docs, "Most libraries and Flask extensions have been
> ported by now and using Flask with Python 3 is generally a smooth ride.".
>
> Thanks for this amazing piece of software.
>
> --
> Magnun Leno
>
>
> Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgadmin-hackers
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Hi Magun,
Please find my comments inline..
On Fri, Jul 8, 2016 at 9:37 PM, Magnun Leno (Pessoal) <magnun.leno@gmail.com> wrote:
Just found another decoding error, this time during server group creation.
What error are you getting?
Can you please share the exact error message to understand the issue properly?
We will need to change too many places by this logic.
I've attached patch for it.
Path attached.
Thanks.
Thanks again.
---
Magnun Leno
Em 07-07-2016 08:39, Ashesh Vashi escreveu:Sure - I will do.On Thu, Jul 7, 2016 at 2:31 PM, Dave Page <dpage@pgadmin.org> wrote:Thanks Magnun.
Ashesh, can you review this please, and check if similar changes are
needed elsewhere?
Thanks.> --
On Wed, Jul 6, 2016 at 7:55 PM, Magnun Leno (Pessoal)
<magnun.leno@gmail.com> wrote:
> Hi All,
>
> Just a small patch to fix server addition/connection errors when using
> non-ascii languages (pt_BR in my case). I haven't delved enough into the
> codebase to be sure, but there might be more occurrences of errmsg that need
> decoding from ASCII to UTF-8.
>
> Is there any plans to port PgAdmin4 to Python3, since Python 2 will only be
> maintained until 2020? Python 3 support was added in Flask 0.10 and, as
> mentioned in the Flask Docs, "Most libraries and Flask extensions have been
> ported by now and using Flask with Python 3 is generally a smooth ride.".
>
> Thanks for this amazing piece of software.
>
> --
> Magnun Leno
>
>
> Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgadmin-hackers
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers
Attachment
On Wed, Jul 13, 2016 at 1:11 AM, Magnun Leno <magnun.leno@gmail.com> wrote:
Sorry, my bad. Here is the traceback:What error are you getting?Can you please share the exact error message to understand the issue properly?
2016-07-12 16:12:22,366: ERROR pgadmin: Exception on /browser/server-group/obj/ [POST]
Traceback (most recent call last):
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/utils.py", line 235, in dispatch_request
return method(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/server_groups/__init__.py", line 204, in create
data = request.form if request.form else json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)
How to reproduce:More info about my env:
- Start PgAdmin4;
- Access localhost:5050
- Login
- In the left panel, right click in Servers. Then, "Create > Server Group", or "Create > Server" (both share the same error)
- In the "Name field", write any non ASCII word. In my case I was using the following name: Produção
- Click "Save"
$ env | grep LANG
LANG=pt_BR.UTF-8
GDM_LANG=pt_BR.UTF-8
$ python --version
Python 2.7.11
$ pip freeze
Babel==1.3
beautifulsoup4==4.4.1
blinker==1.3
django-htmlmin==0.8.0
extras==0.0.3
fixtures==2.0.0
Flask==0.10.1
Flask-Babel==0.9
Flask-Gravatar==0.4.2
Flask-Login==0.2.11
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-Security==1.7.4
Flask-SQLAlchemy==2.0
Flask-WTF==0.11
html5lib==1.0b3
importlib==1.0.3
itsdangerous==0.24
Jinja2==2.7.3
linecache2==1.0.0
MarkupSafe==0.23
passlib==1.6.2
pbr==1.9.1
pgadmin4==1.0b2
psycopg2==2.5.2
pycrypto==2.6.1
pyrsistent==0.11.13
python-dateutil==2.5.0
python-mimeparse==1.5.1
pytz==2014.10
simplejson==3.6.5
six==1.9.0
speaklater==1.3
SQLAlchemy==0.9.8
sqlparse==0.1.19
testscenarios==0.5.0
testtools==2.0.0
traceback2==1.4.0
unittest2==1.1.0
Werkzeug==0.9.6
WTForms==2.0.2I'm not familiar with the code base yet, but not necessarily all occurrences of "decode()" need to be patched, just the ones that deal with user input.We will need to change too many places by this logic.I've attached patch for it.
I've changed only the data coming as the user input for deferent nodes. :-)
I've done some further testing and noticed that even running an insert or update with non ASCII text runs into failure.
I've not yet looked at the other utilities in pgAdmin IV.
I'll look in to it too, and share the updated patch.
This "update users set cidade='Brasília' where id=1;", results in the following traceback:
2016-07-12 16:32:59,964: ERROR pgadmin: Exception on /sqleditor/query_tool/start/2760583 [POST]
Traceback (most recent call last):
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/tools/sqleditor/__init__.py", line 241, in start_query_tool
sql = json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 48: ordinal not in range(128)
But, the other way around isn't true. Executing the update mentioned above in psql and then running a select in PgAdmin4 (select * from users where id=1) works fine.
I'll give it a try with Python 3 as soon as possible, it might solve all the issues since it uses unicode by default.
It's good to have feedback like above.
That will give us the idea, where we need improvements, as all the current developers, working on pgAdmin 4, are from English background.
Thanks for the inputs/feedbacks, please keep them coming.
--
Thanks & Regards,
Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company
Thanks in advance.
---
Magnun Leno
Sorry, my bad. Here is the traceback:What error are you getting?Can you please share the exact error message to understand the issue properly?
2016-07-12 16:12:22,366: ERROR pgadmin: Exception on /browser/server-group/obj/ [POST]
Traceback (most recent call last):
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/utils.py", line 235, in dispatch_request
return method(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/browser/server_groups/__init__.py", line 204, in create
data = request.form if request.form else json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14: ordinal not in range(128)
How to reproduce:
- Start PgAdmin4;
- Access localhost:5050
- Login
- In the left panel, right click in Servers. Then, "Create > Server Group", or "Create > Server" (both share the same error)
- In the "Name field", write any non ASCII word. In my case I was using the following name: Produção
- Click "Save"
$ env | grep LANG
LANG=pt_BR.UTF-8
GDM_LANG=pt_BR.UTF-8
$ python --version
Python 2.7.11
$ pip freeze
Babel==1.3
beautifulsoup4==4.4.1
blinker==1.3
django-htmlmin==0.8.0
extras==0.0.3
fixtures==2.0.0
Flask==0.10.1
Flask-Babel==0.9
Flask-Gravatar==0.4.2
Flask-Login==0.2.11
Flask-Mail==0.9.1
Flask-Principal==0.4.0
Flask-Security==1.7.4
Flask-SQLAlchemy==2.0
Flask-WTF==0.11
html5lib==1.0b3
importlib==1.0.3
itsdangerous==0.24
Jinja2==2.7.3
linecache2==1.0.0
MarkupSafe==0.23
passlib==1.6.2
pbr==1.9.1
pgadmin4==1.0b2
psycopg2==2.5.2
pycrypto==2.6.1
pyrsistent==0.11.13
python-dateutil==2.5.0
python-mimeparse==1.5.1
pytz==2014.10
simplejson==3.6.5
six==1.9.0
speaklater==1.3
SQLAlchemy==0.9.8
sqlparse==0.1.19
testscenarios==0.5.0
testtools==2.0.0
traceback2==1.4.0
unittest2==1.1.0
Werkzeug==0.9.6
WTForms==2.0.2
I'm not familiar with the code base yet, but not necessarily all occurrences of "decode()" need to be patched, just the ones that deal with user input.We will need to change too many places by this logic.I've attached patch for it.
I've done some further testing and noticed that even running an insert or update with non ASCII text runs into failure. This "update users set cidade='Brasília' where id=1;", results in the following traceback:
2016-07-12 16:32:59,964: ERROR pgadmin: Exception on /sqleditor/query_tool/start/2760583 [POST]
Traceback (most recent call last):
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/flask_login.py", line 758, in decorated_view
return func(*args, **kwargs)
File "/home/magnun/venv/pgadmin4/lib/python2.7/site-packages/pgadmin4/pgadmin/tools/sqleditor/__init__.py", line 241, in start_query_tool
sql = json.loads(request.data.decode())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 48: ordinal not in range(128)
But, the other way around isn't true. Executing the update mentioned above in psql and then running a select in PgAdmin4 (select * from users where id=1) works fine.
I'll give it a try with Python 3 as soon as possible, it might solve all the issues since it uses unicode by default.
Thanks in advance.
---
Magnun Leno
Great!I've changed only the data coming as the user input for deferent nodes. :-)
I'm really interested in contributing to the project. I hope to have some spare time soon, so I can study the source code and send some improvements, not just fixes.It's good to have feedback like above.That will give us the idea, where we need improvements, as all the current developers, working on pgAdmin 4, are from English background.
I will. Thanks for being so receptive, generally big open source aren't friendly.Thanks for the inputs/feedbacks, please keep them coming.
---
Magnun Leno