Thread: [GSoC][New Feature] Editable and Read-only Columns

Hi, hackers !

Please find attached a work-in-progress patch adding a new feature to the Query Tool on top of updatable result-sets.

This patch allows individual columns of an updatable result-set to be editable or read-only. This allows for a wider variety of updatable result-sets, for example:

- Result-sets with duplicated columns.

- Result-sets with renamed columns (if a column is renamed to a primary key name, the real primary key can be correctly identified) .

- Result-sets including columns that are not selected directly from a table (e.g concatenation of 2 columns or system columns).

In the above cases, these columns would be read-only while other columns of the result-set are editable. Editable/Read-only columns are identified by icons and tooltips in the column header.

This is still a work-in-progress, updates to tests and documentation is still due. Looking forward to your thoughts and feedback!

Also, do you think the editable/read-only icons should apply in both View/Edit Data and Query Tool for consistency? or hidden from View/Edit Data as all columns are editable anyway?

Thanks.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

Aditya, can you review this please?

Thanks.

On Thu, Aug 15, 2019 at 8:01 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi, hackers !

Please find attached a work-in-progress patch adding a new feature to the Query Tool on top of updatable result-sets.

This patch allows individual columns of an updatable result-set to be editable or read-only. This allows for a wider variety of updatable result-sets, for example:

- Result-sets with duplicated columns.

- Result-sets with renamed columns (if a column is renamed to a primary key name, the real primary key can be correctly identified) .

- Result-sets including columns that are not selected directly from a table (e.g concatenation of 2 columns or system columns).

In the above cases, these columns would be read-only while other columns of the result-set are editable. Editable/Read-only columns are identified by icons and tooltips in the column header.

This is still a work-in-progress, updates to tests and documentation is still due. Looking forward to your thoughts and feedback!

Also, do you think the editable/read-only icons should apply in both View/Edit Data and Query Tool for consistency? or hidden from View/Edit Data as all columns are editable anyway?

Thanks.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

Please find an updated patch attached including fixing mentioned issues, python and feature tests, and updates to docs (including a new screenshot).

On Thu, Aug 22, 2019 at 7:52 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

Nice work there !! :)

Few suggestions:

1) Wrap texts like "tooltip = 'Editable column' " in gettext so that they are translated to language selected.

2) One of the api test case is failing. I tried on PG 9.4, Python 3.5. Tested 2 times.

======================================================================
FAIL: runTest (pgadmin.tools.sqleditor.utils.tests.test_is_query_resultset_updatable.TestQueryUpdatableResultset)
When renaming a column to a primary key name
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/tests/test_is_query_resultset_updatable.py", line 100, in runTest
    self._check_primary_keys(response_data)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/tests/test_is_query_resultset_updatable.py", line 119, in _check_primary_keys
    self.assertEquals(primary_keys, self.primary_keys)
AssertionError: {'pk_col1': 'int4', 'pk_col2': 'int4'} != None

3) For table containing geometry columns, the lock icon is not shown for non-editable columns. Instead, geometry view icon is shown. Also the data type is incorrect. Below is the screenshot. Please note, this seems to be happening only for text columns.

On Wed, Aug 21, 2019 at 4:01 PM Dave Page <dpage@pgadmin.org> wrote:

Aditya, can you review this please?

Thanks.

On Thu, Aug 15, 2019 at 8:01 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi, hackers !

Please find attached a work-in-progress patch adding a new feature to the Query Tool on top of updatable result-sets.

This patch allows individual columns of an updatable result-set to be editable or read-only. This allows for a wider variety of updatable result-sets, for example:

- Result-sets with duplicated columns.

- Result-sets with renamed columns (if a column is renamed to a primary key name, the real primary key can be correctly identified) .

- Result-sets including columns that are not selected directly from a table (e.g concatenation of 2 columns or system columns).

In the above cases, these columns would be read-only while other columns of the result-set are editable. Editable/Read-only columns are identified by icons and tooltips in the column header.

This is still a work-in-progress, updates to tests and documentation is still due. Looking forward to your thoughts and feedback!

Also, do you think the editable/read-only icons should apply in both View/Edit Data and Query Tool for consistency? or hidden from View/Edit Data as all columns are editable anyway?

Thanks.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Dave,

I am now using the same versions of Python, Chrome and chromedriver. The tests still pass every time. Are you sure you applied the latest patch? I re-created the patch as a sanity check, please find it attached.

I tried running the specific test a lot of times, passes every time. The only difference is that I am running Ubuntu, otherwise, everything is the same.

On Fri, Aug 23, 2019 at 5:00 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

Hi Yosry,

It's failing for me as well. I debugged and found that, sometimes the grid render is delayed (in microseconds) after query execution. Because of which, the cell you had selected using xpath is actually the older one. And when you're trying to set the value the cell no longer exists as the new cell is rendered now. That's why stale element exception. Adding below code solved the problem for me. Attached is the patch.

import time
time.sleep(0.5)
for column_index, should_be_editable in enumerated_should_be_editable:

On Sat, Aug 24, 2019 at 4:08 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

I am now using the same versions of Python, Chrome and chromedriver. The tests still pass every time. Are you sure you applied the latest patch? I re-created the patch as a sanity check, please find it attached.

I tried running the specific test a lot of times, passes every time. The only difference is that I am running Ubuntu, otherwise, everything is the same.

On Fri, Aug 23, 2019 at 5:00 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

Hi Aditya,

Thanks a lot. 

Do you have any idea why this problem never occurred on my machine by the way?

On Sat, Aug 24, 2019, 8:18 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

It's failing for me as well. I debugged and found that, sometimes the grid render is delayed (in microseconds) after query execution. Because of which, the cell you had selected using xpath is actually the older one. And when you're trying to set the value the cell no longer exists as the new cell is rendered now. That's why stale element exception. Adding below code solved the problem for me. Attached is the patch.

import time
time.sleep(0.5)
for column_index, should_be_editable in enumerated_should_be_editable:

On Sat, Aug 24, 2019 at 4:08 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

I am now using the same versions of Python, Chrome and chromedriver. The tests still pass every time. Are you sure you applied the latest patch? I re-created the patch as a sanity check, please find it attached.

I tried running the specific test a lot of times, passes every time. The only difference is that I am running Ubuntu, otherwise, everything is the same.

On Fri, Aug 23, 2019 at 5:00 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

Hi Yosry,

My guess is because selenium is running faster on our machine. Depends on machine.

On Sat, Aug 24, 2019, 12:06 Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

Thanks a lot. 

Do you have any idea why this problem never occurred on my machine by the way?

On Sat, Aug 24, 2019, 8:18 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

It's failing for me as well. I debugged and found that, sometimes the grid render is delayed (in microseconds) after query execution. Because of which, the cell you had selected using xpath is actually the older one. And when you're trying to set the value the cell no longer exists as the new cell is rendered now. That's why stale element exception. Adding below code solved the problem for me. Attached is the patch.

import time
time.sleep(0.5)
for column_index, should_be_editable in enumerated_should_be_editable:

On Sat, Aug 24, 2019 at 4:08 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

I am now using the same versions of Python, Chrome and chromedriver. The tests still pass every time. Are you sure you applied the latest patch? I re-created the patch as a sanity check, please find it attached.

I tried running the specific test a lot of times, passes every time. The only difference is that I am running Ubuntu, otherwise, everything is the same.

On Fri, Aug 23, 2019 at 5:00 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

Makes sense. Thanks a lot for your help :)

On Sat, Aug 24, 2019, 8:40 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

My guess is because selenium is running faster on our machine. Depends on machine.

On Sat, Aug 24, 2019, 12:06 Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

Thanks a lot. 

Do you have any idea why this problem never occurred on my machine by the way?

On Sat, Aug 24, 2019, 8:18 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

It's failing for me as well. I debugged and found that, sometimes the grid render is delayed (in microseconds) after query execution. Because of which, the cell you had selected using xpath is actually the older one. And when you're trying to set the value the cell no longer exists as the new cell is rendered now. That's why stale element exception. Adding below code solved the problem for me. Attached is the patch.

import time
time.sleep(0.5)
for column_index, should_be_editable in enumerated_should_be_editable:

On Sat, Aug 24, 2019 at 4:08 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

I am now using the same versions of Python, Chrome and chromedriver. The tests still pass every time. Are you sure you applied the latest patch? I re-created the patch as a sanity check, please find it attached.

I tried running the specific test a lot of times, passes every time. The only difference is that I am running Ubuntu, otherwise, everything is the same.

On Fri, Aug 23, 2019 at 5:00 PM Dave Page <dpage@pgadmin.org> wrote:

Hi

I've run it again 4 more times and it fails consistently for me. I'm running on macOS 10.14.5 with Chrome 76.0.3809.100.

On Fri, Aug 23, 2019 at 3:17 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Dave,

The tests pass on my device normally. I am using Python 3.6 on Ubuntu. I have ran all tests using "make check" and they all succeeded. I also ran feature_tests/query_tool_journey_test.py multiple times on its own using "cd web && python regression/runtests.py --pkg feature_tests --modules query_tool_journey_test" and it passed successfully. Could you try again?

Generally, I have found that StaleElementReferenceException occurs intermittently on various feature tests.

On Fri, Aug 23, 2019 at 3:46 PM Dave Page <dpage@pgadmin.org> wrote:

Thanks. Unfortunately this is failing the tests:

======================================================================
ERROR: runTest (pgadmin.feature_tests.query_tool_journey_test.QueryToolJourneyTest)
Tests the path through the query tool
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 85, in runTest
    self._test_updatable_resultset()
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 254, in _test_updatable_resultset
    discard_changes_modal=True)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 417, in _check_query_results_editable
    is_editable = self._check_cell_editable(column_index)
  File "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_journey_test.py", line 430, in _check_cell_editable
    cell_value = int(cell_el.text)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 76, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 628, in _execute
    return self._parent.execute(command, params)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
    self.error_handler.check_response(response)
  File "/Users/dpage/.virtualenvs/pgadmin4/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=76.0.3809.100)
  (Driver info: chromedriver=76.0.3809.126 (d80a294506b4c9d18015e755cee48f953ddc3f2f-refs/branch-heads/3809@{#1024}),platform=Mac OS X 10.14.5 x86_64)

Without the patch, the tests pass.

On Fri, Aug 23, 2019 at 1:10 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

Please find an updated patch attached.

On Fri, Aug 23, 2019 at 1:22 PM Dave Page <dpage@pgadmin.org> wrote:

Except it now doesn't apply as the async connection/rendering patch was in the queue right ahead of it :-(.

Can I get a rebased patch please?

On Fri, Aug 23, 2019 at 11:57 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Excellent !! Looks good to me.

On Fri, Aug 23, 2019 at 2:23 PM Yosry Muhammad <yosrym93@gmail.com> wrote:

 I updated the patch for backwards compatibility anyway.

On Fri, Aug 23, 2019 at 10:45 AM Dave Page <dpage@pgadmin.org> wrote:

On Fri, Aug 23, 2019 at 9:43 AM Yosry Muhammad <yosrym93@gmail.com> wrote:

Hi Aditya,

On Fri, Aug 23, 2019 at 7:33 AM Aditya Toshniwal <aditya.toshniwal@enterprisedb.com> wrote:

Hi Yosry,

The previously mentioned issues seems to be fixed. Below are few issues:

1) On Python 2.7, I get below error when opening query tool:

2019-08-23 10:49:09,329: ERROR flask.app: Object of type buffer is not JSON serializable
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 1544, in get_query_history
    return QueryHistory.get(current_user.id, trans_obj.sid, conn.db)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/query_history.py", line 21, in get
    'result': [rec.query_info for rec in result]
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 75, in make_json_response
    separators=(',', ':'), encoding=encoding),
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/utils/ajax.py", line 30, in default
    return json.JSONEncoder.default(self, obj)
  File "/Users/adityatoshniwal/venv/pypg27/lib/python2.7/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type buffer is not JSON serializable

This error exists on the master branch on Python 2.7, it is not caused by the patch. It is also not a regression of my previous Query History Enhancement patch, I checked.

 

2) The patch is not compatible with older pyscopg2 (2.7.7). I get below error in View/Edit data.

2019-08-23 10:53:12,020: ERROR flask.app: 'table_column'
Traceback (most recent call last):
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/adityatoshniwal/venv/pypg35/lib/python3.5/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/__init__.py", line 435, in poll
    columns = trans_obj.get_columns_types(conn)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/command.py", line 688, in get_columns_types
    table_oid=table_oid)
  File "/Users/adityatoshniwal/projects/pgadmin4/web/pgadmin/tools/sqleditor/utils/get_column_types.py", line 38, in get_columns_types
    if row['oid'] == col['table_column']:
KeyError: 'table_column'

I recall Dave mentioning that it is okay to use psycopg 2.8+ features and that it was a requirement of pgAdmin 4 now (clearly mentioned in requirements.txt too).

I did - then we had the whole table_oid issue on Debian/Ubuntu :-(

 

However, I updated the patch for backwards compatibility. Please find an updated patch attached.

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Yosry Muhammad Yosry

Computer Engineering student,

The Faculty of Engineering,

Cairo University (2021).

Class representative of CMP 2021.

https://www.linkedin.com/in/yosrym93/

--

Thanks and Regards,

Aditya Toshniwal

Software Engineer | EnterpriseDB India | Pune

"Don't Complain about Heat, Plant a TREE"