Thread: [pgadmin-hackers] Server side cursor limitations for on demand loading of data in querytool [RM2137] [pgAdmin4]

Hi

On Tue, Apr 25, 2017 at 8:41 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

To implement feature #2137 we'll need to use server cursor. However server cursor has some

limitation.

For eg.

1. It cannot execute BEGIN; query (basically user cannot start new database transaction)

2. In case if Auto commit is true then we try to execute user queries inside BEGIN and END when ever it's possible even though user has not put BEGIN and END in his query.

Also not all queries executed using Query tool produces records as result. So can we assume only

queries started with SELECT should be executed using server cursor to support on demand loading.

Or should we give user an option to use on demand loading like we have options for Auto commit? and Auto rollback?

In case of on demand loading option user will be responsible to execute correct queries (queries which can be executed using server cursor)

Let me know your opinion on this.

Hmm, those are good points. 

So, as a first step, there's no absolute requirement to use a server side cursor here. The results can be materialised in libpq/psycopg2 (perhaps using an async query), then transferred to the client in batches as described in the ticket.

I think this would be a significant improvemet - we can re-visit the possibility of using server side cursors in the future when we have more ability to parse the query string before executing it (something we will want to do when we merge query tool/edit grid functionality).

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Dave,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, May 8, 2017 at 4:49 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

This is fantastic! The speedup in the query tool is phenomenal with large result sets. For 96K rows, I'm now seeing 1 second until I can browse the data, vs. 13 in v1.4 (and 22 secs in pgAdmin 3)!! Awesome work Harshal :-)

Questions/comments:

- Can we put the row number in the left-hand column, to the right of the checkbox? This patch highlights just how much value that had in pgAdmin 3 (I hadn't realised we had missed it until now)

- If the user clicks the checkbox to select all rows, we need to retrieve them all at that time, otherwise they may be inadvertently working with a truncated result set.

Sure I'll do above both the changes.

 

- Are any changes needed to ensure the Download button works? I'm seeing missing rows in my test here, but that could be because of the known issues there with Unicode.

This is fixed and checked in.

 

Thanks!

On Sun, May 7, 2017 at 6:10 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Here is patch for initial implementation of on demand loading of result set for query tool and datagrid.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Apr 25, 2017 at 5:21 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Apr 25, 2017 at 8:41 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

To implement feature #2137 we'll need to use server cursor. However server cursor has some

limitation.

For eg.

1. It cannot execute BEGIN; query (basically user cannot start new database transaction)

2. In case if Auto commit is true then we try to execute user queries inside BEGIN and END when ever it's possible even though user has not put BEGIN and END in his query.

Also not all queries executed using Query tool produces records as result. So can we assume only

queries started with SELECT should be executed using server cursor to support on demand loading.

Or should we give user an option to use on demand loading like we have options for Auto commit? and Auto rollback?

In case of on demand loading option user will be responsible to execute correct queries (queries which can be executed using server cursor)

Let me know your opinion on this.

Hmm, those are good points. 

So, as a first step, there's no absolute requirement to use a server side cursor here. The results can be materialised in libpq/psycopg2 (perhaps using an async query), then transferred to the client in batches as described in the ticket.

I think this would be a significant improvemet - we can re-visit the possibility of using server side cursors in the future when we have more ability to parse the query string before executing it (something we will want to do when we merge query tool/edit grid functionality).

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 9, 2017 at 1:08 AM, Sarah McAlear <smcalear@pivotal.io> wrote:

Hi Harshal!

We applied your patch and ran the Javascript tests and realized there are 6 tests failing. After that we replicate the failures by hand and found the following issues:

- Clicking in the checkboxes of the rows does not select the row anymore.

- When copying and pasting (any) data, we realized that it copied an empty dataset.

Thanks for pointing out these issues. I have fixed them at my end and also remaining tests are failing because in earlier implementation of sql editor we have used pain 2Darray to provide data to slick grid and In this patch I have used Slick grid DataView to provide data to slick gird. I'll need to update test cases to support this DataView change. I'll send updated patch with all of the above changes along with suggestion given by Dave.

 

We didn't review the patch further because there aren't any tests for the newly implemented functionality. 

Is there a place to find a dataset with 96k rows that you guys were testing with?

Execute below SQL queries to create table dummy_data  with dataset of 100k records. (you can change number of records to generate by replacing 100000 in below SQL query)

CREATE TABLE public.dummy_data

(

   id serial NOT NULL, 

   name text

) 

WITH (

  OIDS = FALSE

)

;

ALTER TABLE public.dummy_data

  OWNER TO postgres;

INSERT INTO public.dummy_data(

            id, name)

     SELECT generate_series(1,100000), 'dummy name';

 

Note: To run the Javascript test you can use the following commands

```

$ cd web

$ yarn install

$ yarn run karma start --single-run

```

Thanks

Joao & Sarah

On Mon, May 8, 2017 at 9:16 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, May 8, 2017 at 4:49 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

This is fantastic! The speedup in the query tool is phenomenal with large result sets. For 96K rows, I'm now seeing 1 second until I can browse the data, vs. 13 in v1.4 (and 22 secs in pgAdmin 3)!! Awesome work Harshal :-)

Questions/comments:

- Can we put the row number in the left-hand column, to the right of the checkbox? This patch highlights just how much value that had in pgAdmin 3 (I hadn't realised we had missed it until now)

- If the user clicks the checkbox to select all rows, we need to retrieve them all at that time, otherwise they may be inadvertently working with a truncated result set.

Sure I'll do above both the changes.

 

- Are any changes needed to ensure the Download button works? I'm seeing missing rows in my test here, but that could be because of the known issues there with Unicode.

This is fixed and checked in.

 

Thanks!

On Sun, May 7, 2017 at 6:10 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Here is patch for initial implementation of on demand loading of result set for query tool and datagrid.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Apr 25, 2017 at 5:21 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Apr 25, 2017 at 8:41 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

To implement feature #2137 we'll need to use server cursor. However server cursor has some

limitation.

For eg.

1. It cannot execute BEGIN; query (basically user cannot start new database transaction)

2. In case if Auto commit is true then we try to execute user queries inside BEGIN and END when ever it's possible even though user has not put BEGIN and END in his query.

Also not all queries executed using Query tool produces records as result. So can we assume only

queries started with SELECT should be executed using server cursor to support on demand loading.

Or should we give user an option to use on demand loading like we have options for Auto commit? and Auto rollback?

In case of on demand loading option user will be responsible to execute correct queries (queries which can be executed using server cursor)

Let me know your opinion on this.

Hmm, those are good points. 

So, as a first step, there's no absolute requirement to use a server side cursor here. The results can be materialised in libpq/psycopg2 (perhaps using an async query), then transferred to the client in batches as described in the ticket.

I think this would be a significant improvemet - we can re-visit the possibility of using server side cursors in the future when we have more ability to parse the query string before executing it (something we will want to do when we merge query tool/edit grid functionality).

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

Pls find updated patch for on demand loading feature.

Changes:

1. Added row number.

2. load renaming result set if user performs select all operation either on grid or column.

3. Fixed Row selection issue and row past issue.

4. Updated existing jasmine test cases.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 9, 2017 at 10:02 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 9, 2017 at 1:08 AM, Sarah McAlear <smcalear@pivotal.io> wrote:

Hi Harshal!

We applied your patch and ran the Javascript tests and realized there are 6 tests failing. After that we replicate the failures by hand and found the following issues:

- Clicking in the checkboxes of the rows does not select the row anymore.

- When copying and pasting (any) data, we realized that it copied an empty dataset.

Thanks for pointing out these issues. I have fixed them at my end and also remaining tests are failing because in earlier implementation of sql editor we have used pain 2Darray to provide data to slick grid and In this patch I have used Slick grid DataView to provide data to slick gird. I'll need to update test cases to support this DataView change. I'll send updated patch with all of the above changes along with suggestion given by Dave.

 

We didn't review the patch further because there aren't any tests for the newly implemented functionality. 

Is there a place to find a dataset with 96k rows that you guys were testing with?

Execute below SQL queries to create table dummy_data  with dataset of 100k records. (you can change number of records to generate by replacing 100000 in below SQL query)

CREATE TABLE public.dummy_data

(

   id serial NOT NULL, 

   name text

) 

WITH (

  OIDS = FALSE

)

;

ALTER TABLE public.dummy_data

  OWNER TO postgres;

INSERT INTO public.dummy_data(

            id, name)

     SELECT generate_series(1,100000), 'dummy name';

 

Note: To run the Javascript test you can use the following commands

```

$ cd web

$ yarn install

$ yarn run karma start --single-run

```

Thanks

Joao & Sarah

On Mon, May 8, 2017 at 9:16 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, May 8, 2017 at 4:49 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

This is fantastic! The speedup in the query tool is phenomenal with large result sets. For 96K rows, I'm now seeing 1 second until I can browse the data, vs. 13 in v1.4 (and 22 secs in pgAdmin 3)!! Awesome work Harshal :-)

Questions/comments:

- Can we put the row number in the left-hand column, to the right of the checkbox? This patch highlights just how much value that had in pgAdmin 3 (I hadn't realised we had missed it until now)

- If the user clicks the checkbox to select all rows, we need to retrieve them all at that time, otherwise they may be inadvertently working with a truncated result set.

Sure I'll do above both the changes.

 

- Are any changes needed to ensure the Download button works? I'm seeing missing rows in my test here, but that could be because of the known issues there with Unicode.

This is fixed and checked in.

 

Thanks!

On Sun, May 7, 2017 at 6:10 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Here is patch for initial implementation of on demand loading of result set for query tool and datagrid.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Apr 25, 2017 at 5:21 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Apr 25, 2017 at 8:41 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

To implement feature #2137 we'll need to use server cursor. However server cursor has some

limitation.

For eg.

1. It cannot execute BEGIN; query (basically user cannot start new database transaction)

2. In case if Auto commit is true then we try to execute user queries inside BEGIN and END when ever it's possible even though user has not put BEGIN and END in his query.

Also not all queries executed using Query tool produces records as result. So can we assume only

queries started with SELECT should be executed using server cursor to support on demand loading.

Or should we give user an option to use on demand loading like we have options for Auto commit? and Auto rollback?

In case of on demand loading option user will be responsible to execute correct queries (queries which can be executed using server cursor)

Let me know your opinion on this.

Hmm, those are good points. 

So, as a first step, there's no absolute requirement to use a server side cursor here. The results can be materialised in libpq/psycopg2 (perhaps using an async query), then transferred to the client in batches as described in the ticket.

I think this would be a significant improvemet - we can re-visit the possibility of using server side cursors in the future when we have more ability to parse the query string before executing it (something we will want to do when we merge query tool/edit grid functionality).

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

Here is rebased patch.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, May 12, 2017 at 1:28 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Pls find updated patch for on demand loading feature.

Changes:

1. Added row number.

2. load renaming result set if user performs select all operation either on grid or column.

3. Fixed Row selection issue and row past issue.

4. Updated existing jasmine test cases.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 9, 2017 at 10:02 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 9, 2017 at 1:08 AM, Sarah McAlear <smcalear@pivotal.io> wrote:

Hi Harshal!

We applied your patch and ran the Javascript tests and realized there are 6 tests failing. After that we replicate the failures by hand and found the following issues:

- Clicking in the checkboxes of the rows does not select the row anymore.

- When copying and pasting (any) data, we realized that it copied an empty dataset.

Thanks for pointing out these issues. I have fixed them at my end and also remaining tests are failing because in earlier implementation of sql editor we have used pain 2Darray to provide data to slick grid and In this patch I have used Slick grid DataView to provide data to slick gird. I'll need to update test cases to support this DataView change. I'll send updated patch with all of the above changes along with suggestion given by Dave.

 

We didn't review the patch further because there aren't any tests for the newly implemented functionality. 

Is there a place to find a dataset with 96k rows that you guys were testing with?

Execute below SQL queries to create table dummy_data  with dataset of 100k records. (you can change number of records to generate by replacing 100000 in below SQL query)

CREATE TABLE public.dummy_data

(

   id serial NOT NULL, 

   name text

) 

WITH (

  OIDS = FALSE

)

;

ALTER TABLE public.dummy_data

  OWNER TO postgres;

INSERT INTO public.dummy_data(

            id, name)

     SELECT generate_series(1,100000), 'dummy name';

 

Note: To run the Javascript test you can use the following commands

```

$ cd web

$ yarn install

$ yarn run karma start --single-run

```

Thanks

Joao & Sarah

On Mon, May 8, 2017 at 9:16 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, May 8, 2017 at 4:49 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

This is fantastic! The speedup in the query tool is phenomenal with large result sets. For 96K rows, I'm now seeing 1 second until I can browse the data, vs. 13 in v1.4 (and 22 secs in pgAdmin 3)!! Awesome work Harshal :-)

Questions/comments:

- Can we put the row number in the left-hand column, to the right of the checkbox? This patch highlights just how much value that had in pgAdmin 3 (I hadn't realised we had missed it until now)

- If the user clicks the checkbox to select all rows, we need to retrieve them all at that time, otherwise they may be inadvertently working with a truncated result set.

Sure I'll do above both the changes.

 

- Are any changes needed to ensure the Download button works? I'm seeing missing rows in my test here, but that could be because of the known issues there with Unicode.

This is fixed and checked in.

 

Thanks!

On Sun, May 7, 2017 at 6:10 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Here is patch for initial implementation of on demand loading of result set for query tool and datagrid.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Apr 25, 2017 at 5:21 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Tue, Apr 25, 2017 at 8:41 AM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi Dave,

To implement feature #2137 we'll need to use server cursor. However server cursor has some

limitation.

For eg.

1. It cannot execute BEGIN; query (basically user cannot start new database transaction)

2. In case if Auto commit is true then we try to execute user queries inside BEGIN and END when ever it's possible even though user has not put BEGIN and END in his query.

Also not all queries executed using Query tool produces records as result. So can we assume only

queries started with SELECT should be executed using server cursor to support on demand loading.

Or should we give user an option to use on demand loading like we have options for Auto commit? and Auto rollback?

In case of on demand loading option user will be responsible to execute correct queries (queries which can be executed using server cursor)

Let me know your opinion on this.

Hmm, those are good points. 

So, as a first step, there's no absolute requirement to use a server side cursor here. The results can be materialised in libpq/psycopg2 (perhaps using an async query), then transferred to the client in batches as described in the ticket.

I think this would be a significant improvemet - we can re-visit the possibility of using server side cursors in the future when we have more ability to parse the query string before executing it (something we will want to do when we merge query tool/edit grid functionality).

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

On Mon, May 15, 2017 at 7:40 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

On Sat, May 13, 2017 at 12:35 AM, Joao Pedro De Almeida Pereira <jdealmeidapereira@pivotal.io> wrote:

We were only able to apply the patch on 1f903ba2 (were seeing patch does not apply due to sqleditor.js conflicts)

The javascript tests passed, but we were unable to copy rows or columns or cells when running the application. Could you run feature tests?

There are three modes sqleditor can be launched

1. Query tool  (Tools menus -> Query Tool)

2. Datagrid.  (Right click on any table/view  -> View Data -> View All/First 100/Last 100/Filtered rows)

3. Scripts (Right click on any table/view -> INSERT/CREATE/UPDATE/DELETE/SELECT)

Paste functionality is only enabled in Datagrid and table has Primary key otherwise it's disabled. In your case row might have been copied but you were unable paste because you might be trying to paste the rows in Query tool. Please try again in Datagrid with table having Primary key.

 

Now that more functionality is being added to sqleditor.js, this may be a good time to extract the functionality to separate files. This will increase readability, and encourage separation of concerns. It will also make changes easier to test in isolation.

Ok. Let me check if I can separate out ant functionalities.

It's probably a good idea to test the changes made to the python as well as javascript code. In this case, the new behavior of poll() in sqleditor __init__ should be tested.

At this point we don't have any python unit tests that can test sqleditor backend (python code).

@Dave should I include python unit test cases in this patch?

We do have some feature tests that hit the query tool - Murtuza wrote some anti-XSS validation tests for example, and Khushboo has been working on some datatype rendering tests.

As a general rule, I prefer we focus more on feature tests now than the API tests - they cover the whole app end-to-end of course. The disadvantages are:

- The treeview isn't reliable enough for me to enable those tests on the CI server yet.

- They can take a long time to run, so we need to test multiple things at once wherever possible. That means minimising browser reloads, or new instances of tools like the Query Tool - or even the number of queries executed through the query tool as part of a test.

That said, yes, if there are specific things that are not covered by Murtuza and Khushboo's work, we should test them. For example, loading all rows when the user selects all, running/rendering EXPLAIN, auto-commit vs. auto-rollback (and combinations thereof).

The standard moving forwards should be to include feature tests for new functionality and Jasmine tests for algorithmic JS code.

I also agree with Joao on the modularisation of JS code. Testable and reusable code should be in "library" files, and we should work to minimise the amount of JS templates - for the most part, that means moving to the client-side translation mechanism which Tira worked on, and I've done some early migration work.

Thanks.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi,

Please find attached updated patch for feature RM2137.

Changes in this patch:

1. Patch rebased.

2. Updated existing feature tests which requires changes due to this feature.

     affected feature test cases:

     i. PGDataypeFeatureTest

     ii. CheckForXssFeatureTest

3. Updated existing jasmine test cases which requires changes due to this feature.

     affected jasmine test cases:

     i. copy data

     ii. range_boundary_navigator

     iii. row_selector

     iv. set_stages_rows

4. New feature tests added

    i. on demand result set on scrolling.

    ii. on demand result set on grid select all.

    iii. on demand result set on column select all.

    iv. explain query

    v. explain query with verbose

    vi. explain query with costs

    vii. explain analyze query

    viii. explain analyze query with buffers

    ix. explain analyze query with timing

    x. auto commit disabled.

    xi. auto commit enabled.

    xii. auto rollback enabled.

    xiii. cancel query.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 16, 2017 at 8:14 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, May 15, 2017 at 7:40 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

On Sat, May 13, 2017 at 12:35 AM, Joao Pedro De Almeida Pereira <jdealmeidapereira@pivotal.io> wrote:

We were only able to apply the patch on 1f903ba2 (were seeing patch does not apply due to sqleditor.js conflicts)

The javascript tests passed, but we were unable to copy rows or columns or cells when running the application. Could you run feature tests?

There are three modes sqleditor can be launched

1. Query tool  (Tools menus -> Query Tool)

2. Datagrid.  (Right click on any table/view  -> View Data -> View All/First 100/Last 100/Filtered rows)

3. Scripts (Right click on any table/view -> INSERT/CREATE/UPDATE/DELETE/SELECT)

Paste functionality is only enabled in Datagrid and table has Primary key otherwise it's disabled. In your case row might have been copied but you were unable paste because you might be trying to paste the rows in Query tool. Please try again in Datagrid with table having Primary key.

 

Now that more functionality is being added to sqleditor.js, this may be a good time to extract the functionality to separate files. This will increase readability, and encourage separation of concerns. It will also make changes easier to test in isolation.

Ok. Let me check if I can separate out ant functionalities.

It's probably a good idea to test the changes made to the python as well as javascript code. In this case, the new behavior of poll() in sqleditor __init__ should be tested.

At this point we don't have any python unit tests that can test sqleditor backend (python code).

@Dave should I include python unit test cases in this patch?

We do have some feature tests that hit the query tool - Murtuza wrote some anti-XSS validation tests for example, and Khushboo has been working on some datatype rendering tests.

As a general rule, I prefer we focus more on feature tests now than the API tests - they cover the whole app end-to-end of course. The disadvantages are:

- The treeview isn't reliable enough for me to enable those tests on the CI server yet.

- They can take a long time to run, so we need to test multiple things at once wherever possible. That means minimising browser reloads, or new instances of tools like the Query Tool - or even the number of queries executed through the query tool as part of a test.

That said, yes, if there are specific things that are not covered by Murtuza and Khushboo's work, we should test them. For example, loading all rows when the user selects all, running/rendering EXPLAIN, auto-commit vs. auto-rollback (and combinations thereof).

The standard moving forwards should be to include feature tests for new functionality and Jasmine tests for algorithmic JS code.

I also agree with Joao on the modularisation of JS code. Testable and reusable code should be in "library" files, and we should work to minimise the amount of JS templates - for the most part, that means moving to the client-side translation mechanism which Tira worked on, and I've done some early migration work.

Thanks.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Ignore this patch.

Rebase and migration of feature tests and jasmine tests required.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Jun 8, 2017 at 3:56 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

Please find attached updated patch for feature RM2137.

Changes in this patch:

1. Patch rebased.

2. Updated existing feature tests which requires changes due to this feature.

     affected feature test cases:

     i. PGDataypeFeatureTest

     ii. CheckForXssFeatureTest

3. Updated existing jasmine test cases which requires changes due to this feature.

     affected jasmine test cases:

     i. copy data

     ii. range_boundary_navigator

     iii. row_selector

     iv. set_stages_rows

4. New feature tests added

    i. on demand result set on scrolling.

    ii. on demand result set on grid select all.

    iii. on demand result set on column select all.

    iv. explain query

    v. explain query with verbose

    vi. explain query with costs

    vii. explain analyze query

    viii. explain analyze query with buffers

    ix. explain analyze query with timing

    x. auto commit disabled.

    xi. auto commit enabled.

    xii. auto rollback enabled.

    xiii. cancel query.

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, May 16, 2017 at 8:14 PM, Dave Page <dpage@pgadmin.org> wrote:

On Mon, May 15, 2017 at 7:40 PM, Harshal Dhumal <harshal.dhumal@enterprisedb.com> wrote:

Hi,

On Sat, May 13, 2017 at 12:35 AM, Joao Pedro De Almeida Pereira <jdealmeidapereira@pivotal.io> wrote:

We were only able to apply the patch on 1f903ba2 (were seeing patch does not apply due to sqleditor.js conflicts)

The javascript tests passed, but we were unable to copy rows or columns or cells when running the application. Could you run feature tests?

There are three modes sqleditor can be launched

1. Query tool  (Tools menus -> Query Tool)

2. Datagrid.  (Right click on any table/view  -> View Data -> View All/First 100/Last 100/Filtered rows)

3. Scripts (Right click on any table/view -> INSERT/CREATE/UPDATE/DELETE/SELECT)

Paste functionality is only enabled in Datagrid and table has Primary key otherwise it's disabled. In your case row might have been copied but you were unable paste because you might be trying to paste the rows in Query tool. Please try again in Datagrid with table having Primary key.

 

Now that more functionality is being added to sqleditor.js, this may be a good time to extract the functionality to separate files. This will increase readability, and encourage separation of concerns. It will also make changes easier to test in isolation.

Ok. Let me check if I can separate out ant functionalities.

It's probably a good idea to test the changes made to the python as well as javascript code. In this case, the new behavior of poll() in sqleditor __init__ should be tested.

At this point we don't have any python unit tests that can test sqleditor backend (python code).

@Dave should I include python unit test cases in this patch?

We do have some feature tests that hit the query tool - Murtuza wrote some anti-XSS validation tests for example, and Khushboo has been working on some datatype rendering tests.

As a general rule, I prefer we focus more on feature tests now than the API tests - they cover the whole app end-to-end of course. The disadvantages are:

- The treeview isn't reliable enough for me to enable those tests on the CI server yet.

- They can take a long time to run, so we need to test multiple things at once wherever possible. That means minimising browser reloads, or new instances of tools like the Query Tool - or even the number of queries executed through the query tool as part of a test.

That said, yes, if there are specific things that are not covered by Murtuza and Khushboo's work, we should test them. For example, loading all rows when the user selects all, running/rendering EXPLAIN, auto-commit vs. auto-rollback (and combinations thereof).

The standard moving forwards should be to include feature tests for new functionality and Jasmine tests for algorithmic JS code.

I also agree with Joao on the modularisation of JS code. Testable and reusable code should be in "library" files, and we should work to minimise the amount of JS templates - for the most part, that means moving to the client-side translation mechanism which Tira worked on, and I've done some early migration work.

Thanks.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Jun 14, 2017 at 6:21 PM, Ashesh Vashi <ashesh.vashi@enterprisedb.com> wrote:

On Wed, Jun 14, 2017 at 6:19 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi,

Sorry - it's drifted out again, I suspect because of the work Ashesh
has been doing. Can you rebase please? Check with Ashesh first though,
in case he's about ready to commit another big change.

I am not. :-)

Sure, I'll send updated patch.

 

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company

http://www.linkedin.com/in/asheshvashi 

Thanks.

On Fri, Jun 9, 2017 at 10:08 AM, Harshal Dhumal

<harshal.dhumal@enterprisedb.com> wrote:
> Hi,
>
>
> Please find rebased patch
>
> --
> Harshal Dhumal
> Sr. Software Engineer
>
> EnterpriseDB India: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
> On Thu, Jun 8, 2017 at 6:40 PM, Harshal Dhumal
> <harshal.dhumal@enterprisedb.com> wrote:
>>
>> Ignore this patch.
>> Rebase and migration of feature tests and jasmine tests required.
>>
>> --
>> Harshal Dhumal
>> Sr. Software Engineer
>>
>> EnterpriseDB India: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>> On Thu, Jun 8, 2017 at 3:56 PM, Harshal Dhumal
>> <harshal.dhumal@enterprisedb.com> wrote:
>>>
>>> Hi,
>>> Please find attached updated patch for feature RM2137.
>>>
>>> Changes in this patch:
>>> 1. Patch rebased.
>>>
>>> 2. Updated existing feature tests which requires changes due to this
>>> feature.
>>>      affected feature test cases:
>>>      i. PGDataypeFeatureTest
>>>      ii. CheckForXssFeatureTest
>>>
>>> 3. Updated existing jasmine test cases which requires changes due to this
>>> feature.
>>>      affected jasmine test cases:
>>>      i. copy data
>>>      ii. range_boundary_navigator
>>>      iii. row_selector
>>>      iv. set_stages_rows
>>>
>>> 4. New feature tests added
>>>     i. on demand result set on scrolling.
>>>     ii. on demand result set on grid select all.
>>>     iii. on demand result set on column select all.
>>>     iv. explain query
>>>     v. explain query with verbose
>>>     vi. explain query with costs
>>>     vii. explain analyze query
>>>     viii. explain analyze query with buffers
>>>     ix. explain analyze query with timing
>>>     x. auto commit disabled.
>>>     xi. auto commit enabled.
>>>     xii. auto rollback enabled.
>>>     xiii. cancel query.
>>>
>>>
>>>
>>> --
>>> Harshal Dhumal
>>> Sr. Software Engineer
>>>
>>> EnterpriseDB India: http://www.enterprisedb.com
>>> The Enterprise PostgreSQL Company
>>>
>>> On Tue, May 16, 2017 at 8:14 PM, Dave Page <dpage@pgadmin.org> wrote:
>>>>
>>>>
>>>>
>>>> On Mon, May 15, 2017 at 7:40 PM, Harshal Dhumal
>>>> <harshal.dhumal@enterprisedb.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On Sat, May 13, 2017 at 12:35 AM, Joao Pedro De Almeida Pereira
>>>>> <jdealmeidapereira@pivotal.io> wrote:
>>>>>>
>>>>>> We were only able to apply the patch on 1f903ba2 (were seeing patch
>>>>>> does not apply due to sqleditor.js conflicts)
>>>>>> The javascript tests passed, but we were unable to copy rows or
>>>>>> columns or cells when running the application. Could you run feature tests?
>>>>>
>>>>> There are three modes sqleditor can be launched
>>>>> 1. Query tool  (Tools menus -> Query Tool)
>>>>> 2. Datagrid.  (Right click on any table/view  -> View Data -> View
>>>>> All/First 100/Last 100/Filtered rows)
>>>>> 3. Scripts (Right click on any table/view ->
>>>>> INSERT/CREATE/UPDATE/DELETE/SELECT)
>>>>>
>>>>> Paste functionality is only enabled in Datagrid and table has Primary
>>>>> key otherwise it's disabled. In your case row might have been copied but you
>>>>> were unable paste because you might be trying to paste the rows in Query
>>>>> tool. Please try again in Datagrid with table having Primary key.
>>>>>
>>>>>>
>>>>>>
>>>>>> Now that more functionality is being added to sqleditor.js, this may
>>>>>> be a good time to extract the functionality to separate files. This will
>>>>>> increase readability, and encourage separation of concerns. It will also
>>>>>> make changes easier to test in isolation.
>>>>>
>>>>> Ok. Let me check if I can separate out ant functionalities.
>>>>>
>>>>>>
>>>>>> It's probably a good idea to test the changes made to the python as
>>>>>> well as javascript code. In this case, the new behavior of poll() in
>>>>>> sqleditor __init__ should be tested.
>>>>>
>>>>>
>>>>> At this point we don't have any python unit tests that can test
>>>>> sqleditor backend (python code).
>>>>> @Dave should I include python unit test cases in this patch?
>>>>
>>>>
>>>> We do have some feature tests that hit the query tool - Murtuza wrote
>>>> some anti-XSS validation tests for example, and Khushboo has been working on
>>>> some datatype rendering tests.
>>>>
>>>> As a general rule, I prefer we focus more on feature tests now than the
>>>> API tests - they cover the whole app end-to-end of course. The disadvantages
>>>> are:
>>>>
>>>> - The treeview isn't reliable enough for me to enable those tests on the
>>>> CI server yet.
>>>>
>>>> - They can take a long time to run, so we need to test multiple things
>>>> at once wherever possible. That means minimising browser reloads, or new
>>>> instances of tools like the Query Tool - or even the number of queries
>>>> executed through the query tool as part of a test.
>>>>
>>>> That said, yes, if there are specific things that are not covered by
>>>> Murtuza and Khushboo's work, we should test them. For example, loading all
>>>> rows when the user selects all, running/rendering EXPLAIN, auto-commit vs.
>>>> auto-rollback (and combinations thereof).
>>>>
>>>> The standard moving forwards should be to include feature tests for new
>>>> functionality and Jasmine tests for algorithmic JS code.
>>>>
>>>> I also agree with Joao on the modularisation of JS code. Testable and
>>>> reusable code should be in "library" files, and we should work to minimise
>>>> the amount of JS templates - for the most part, that means moving to the
>>>> client-side translation mechanism which Tira worked on, and I've done some
>>>> early migration work.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> --
>>>> Dave Page
>>>> Blog: http://pgsnake.blogspot.com
>>>> Twitter: @pgsnake
>>>>
>>>> EnterpriseDB UK: http://www.enterprisedb.com
>>>> The Enterprise PostgreSQL Company
>>>
>>>
>>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

--
Sent via pgadmin-hackers mailing list (pgadmin-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgadmin-hackers

Sounds good, thanks.

On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal

<harshal.dhumal@enterprisedb.com> wrote:
> Hi
>
> On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi
>>
>> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
>> <harshal.dhumal@enterprisedb.com> wrote:
>> > Hi Dave,
>> >
>> > Please find attached updated patch.
>> >
>> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage@pgadmin.org> wrote:
>> >>
>> >> Hi
>> >>
>> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
>> >> <harshal.dhumal@enterprisedb.com> wrote:
>> >>>
>> >>> Hi Dave,
>> >>>
>> >>> Please find rebased patch for RM2137.
>> >>
>> >>
>> >> Looking very good. The only issues I see are:
>> >>
>> >> - The row headers should auto-size such that they can display the row
>> >> numbers if the last row was displayed. E.g. if there are 12345 rows in
>> >> total, then the row header should be sized to display 5 digits.
>> >>
>> > Fixed.
>> >
>> >
>> >>
>> >> - The tests are comprehensive, which is awesome. However, every time I
>> >> ran
>> >> them, at least one of the feature tests failed. Unfortunately, it was a
>> >> different one each time. In the last two runs, I got:
>> >>
>> >>  ======================================================================
>> >> ERROR: runTest
>> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> Query tool feature test
>> >> ----------------------------------------------------------------------
>> >> Traceback (most recent call last):
>> >>   File
>> >>
>> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/base_feature_test.py",
>> >> line 40, in setUp
>> >>     self.before()
>> >>   File
>> >>
>> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> line 40, in before
>> >>     self._connects_to_server()
>> >>   File
>> >>
>> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> line 144, in _connects_to_server
>> >>     self.page.driver.find_element_by_link_text("Create"))\
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 319, in find_element_by_link_text
>> >>     return self.find_element(by=By.LINK_TEXT, value=link_text)
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 756, in find_element
>> >>     'value': value})['value']
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 238, in execute
>> >>     self.error_handler.check_response(response)
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >> line 193, in check_response
>> >>     raise exception_class(message, screen, stacktrace)
>> >> NoSuchElementException: Message: no such element: Unable to locate
>> >> element: {"method":"link text","selector":"Create"}
>> >>   (Session info: chrome=58.0.3029.110)
>> >>   (Driver info: chromedriver=2.29.461585
>> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> >> x86_64)
>> >>
>> > I checked code from _connects_to_server function which is common in all
>> > features test cases. I didn't find anything wrong with this. If there is
>> > a
>> > bug in this function then all feature test must fail.
>> > Let me know if you are getting failure consistently in
>> > _connects_to_server
>> > function.
>>
>> I wondered if that one is a race condition. Do we need a short delay
>> before clicking the Object menu? I have seen this occasionally before.
>
>
> OK. In that case let's try putting 1-2 second delay and observer behaviour.
> I'll send separate patch for this tomorrow as this is not related to on
> demand query result feature or its test cases.
>
>>
>>
>> >>
>> >>
>> >> ======================================================================
>> >> ERROR: runTest
>> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> Query tool feature test
>> >> ----------------------------------------------------------------------
>> >> Traceback (most recent call last):
>> >>   File
>> >>
>> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> line 119, in runTest
>> >>     self._query_tool_auto_rollback_enabled()
>> >>   File
>> >>
>> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >> line 697, in _query_tool_auto_rollback_enabled
>> >>     '//div[contains(@class, "sql-editor-message") and
>> >> contains(string(),
>> >> "COMMIT")]'
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 295, in find_element_by_xpath
>> >>     return self.find_element(by=By.XPATH, value=xpath)
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 756, in find_element
>> >>     'value': value})['value']
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >> line 238, in execute
>> >>     self.error_handler.check_response(response)
>> >>   File
>> >>
>> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >> line 193, in check_response
>> >>     raise exception_class(message, screen, stacktrace)
>> >> NoSuchElementException: Message: no such element: Unable to locate
>> >> element: {"method":"xpath","selector":"//div[contains(@class,
>> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
>> >>   (Session info: chrome=58.0.3029.110)
>> >>   (Driver info: chromedriver=2.29.461585
>> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> >> x86_64)
>> >>
>> > I have updated Auto rollback enabled test in this patch.
>> >
>> >>
>> >> Relevant screenshots attached.
>> >>
>> >> - Can you tidy up the regression output a little please? Instead of:
>> >>
>> >> -------
>> >> runTest (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> Query tool feature test ... On demand result set on scrolling...
>> >> OK.
>> >>
>> >> On demand result set on grid select all...
>> >> OK.
>> >>
>> >> On demand result set on column select all...
>> >> OK.
>> >>
>> >> Explain query...
>> >> OK.
>> >>
>> >> Explain query with verbose...
>> >> OK.
>> >>
>> >> Explain query with costs...
>> >> OK.
>> >>
>> >> Explain analyze query...
>> >> OK.
>> >>
>> >> Explain analyze query with buffers...
>> >> OK.
>> >>
>> >> Explain analyze query with timing...
>> >> OK.
>> >>
>> >> Auto commit disabled...
>> >> OK.
>> >>
>> >> Auto commit enabled...
>> >> OK.
>> >>
>> >> Auto rollback enabled...
>> >> ERROR
>> >> -------
>> >>
>> >> Something like:
>> >>
>> >> -------
>> >> runTest (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >> Query tool feature test ...
>> >> On demand result set on scrolling... OK.
>> >> On demand result set on grid select all... OK.
>> >> On demand result set on column select all... OK.
>> >> Explain query... OK.
>> >> Explain query with verbose... OK.
>> >> Explain query with costs... OK.
>> >> Explain analyze query... OK.
>> >> Explain analyze query with buffers... OK.
>> >> Explain analyze query with timing... OK.
>> >> Auto commit disabled... OK.
>> >> Auto commit enabled... OK.
>> >> Auto rollback enabled... ERROR
>> >> --------
>> >>
>> > Fixed.
>> >
>> >
>> >>
>> >> Thanks!
>> >>
>> >> --
>> >> Dave Page
>> >> Blog: http://pgsnake.blogspot.com
>> >> Twitter: @pgsnake
>> >>
>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >
>> >
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

I'm sorry, it needs rebasing again. If you can do it quickly, I'll
make sure it's the next patch I work on in that area.

Thanks.

On Mon, Jun 26, 2017 at 5:16 AM, Harshal Dhumal

<harshal.dhumal@enterprisedb.com> wrote:
> Hi Dave,
>
> Please find updated rebased patch for RM2137
>
> On Fri, Jun 23, 2017 at 9:00 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi Harshal,
>>
>> When can we expect an updated version of this patch? I think it's
>> important to get this into the next release.
>>
>> Thanks!
>>
>> On Fri, Jun 16, 2017 at 10:55 AM, Dave Page <dpage@pgadmin.org> wrote:
>> > Hi,
>> >
>> > That's better - the failures are far less random now :-). I got the
>> > following two though, on both PG and EPAS 9.5:
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> > Query tool feature test
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 95, in runTest
>> >     self._query_tool_explain_analyze_buffers()
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 443, in _query_tool_explain_analyze_buffers
>> >     canvas.find_element_by_xpath("//*[contains(string(), 'Shared Read
>> > Blocks')]")
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 260, in find_element_by_xpath
>> >     return self.find_element(by=By.XPATH, value=xpath)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 508, in find_element
>> >     {"using": by, "value": value})['value']
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> >     return self._parent.execute(command, params)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> >     self.error_handler.check_response(response)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> >     raise exception_class(message, screen, stacktrace)
>> > NoSuchElementException: Message: no such element: Unable to locate
>> > element: {"method":"xpath","selector":"//*[contains(string(), 'Shared
>> > Read Blocks')]"}
>> >   (Session info: chrome=58.0.3029.110)
>> >   (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 104, in runTest
>> >     self._add_row()
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 255, in _add_row
>> >     self._update_cell(cell_xpath, config_data[str(idx)])
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 164, in _update_cell
>> >     cell_el = self.page.find_by_xpath(xpath)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 122, in find_by_xpath
>> >     return self.wait_for_element(lambda driver:
>> > driver.find_element_by_xpath(xpath))
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 205, in wait_for_element
>> >     return self._wait_for("element to exist", element_if_it_exists)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 255, in _wait_for
>> >     "Timed out waiting for " + waiting_for_message)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/support/wait.py",
>> > line 71, in until
>> >     value = method(self._driver)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 200, in element_if_it_exists
>> >     if element.is_displayed() and element.is_enabled():
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 358, in is_displayed
>> >     return self._execute(Command.IS_ELEMENT_DISPLAYED)['value']
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> >     return self._parent.execute(command, params)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> >     self.error_handler.check_response(response)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> >     raise exception_class(message, screen, stacktrace)
>> > StaleElementReferenceException: Message: stale element reference:
>> > element is not attached to the page document
>> >   (Session info: chrome=58.0.3029.110)
>> >   (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>
>
> I checked my feature test cases for any database server version/type (PG,
> EPAS) specific failures and I found that test cases are working fine on all
> of them. However I have slightly modified test case to overcome above
> mentioned failures.
>
>>
>> >
>> > On 9.4 everything passes. On 9.6, I get one failure:
>> >
>> > ======================================================================
>> > FAIL: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 105, in runTest
>> >     self._verify_row_data(True)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 282, in _verify_row_data
>> >     self.assertEquals(cells[idx], config_data[str(idx)][1])
>> > AssertionError: u'[default]' != u'1'
>> > - [default]
>> > + 1
>> >
>
> I guess Khushboo has fixed this recently.
>
>
>>
>> > Screenshots attached.
>> >
>> > On Fri, Jun 16, 2017 at 7:54 AM, Harshal Dhumal
>> > <harshal.dhumal@enterprisedb.com> wrote:
>> >> Hi Dave,
>> >>
>> >> Please find attached patch where I have added timeout of 2 seconds
>> >> before
>> >> selecting object menu in connect to server function.
>> >>
>> >> Note: Apply this patch on top of previous patch for on demand loading
>> >> feature.
>> >>
>> >>
>> >> --
>> >> Harshal Dhumal
>> >> Sr. Software Engineer
>> >>
>> >> EnterpriseDB India: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >>
>> >> On Fri, Jun 16, 2017 at 2:25 AM, Dave Page <dpage@pgadmin.org> wrote:
>> >>>
>> >>> Sounds good, thanks.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal
>> >>> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> > Hi
>> >>> >
>> >>> > On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage@pgadmin.org>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi
>> >>> >>
>> >>> >> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
>> >>> >> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> >> > Hi Dave,
>> >>> >> >
>> >>> >> > Please find attached updated patch.
>> >>> >> >
>> >>> >> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage@pgadmin.org>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi
>> >>> >> >>
>> >>> >> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
>> >>> >> >> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> >> >>>
>> >>> >> >>> Hi Dave,
>> >>> >> >>>
>> >>> >> >>> Please find rebased patch for RM2137.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Looking very good. The only issues I see are:
>> >>> >> >>
>> >>> >> >> - The row headers should auto-size such that they can display
>> >>> >> >> the
>> >>> >> >> row
>> >>> >> >> numbers if the last row was displayed. E.g. if there are 12345
>> >>> >> >> rows
>> >>> >> >> in
>> >>> >> >> total, then the row header should be sized to display 5 digits.
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> - The tests are comprehensive, which is awesome. However, every
>> >>> >> >> time
>> >>> >> >> I
>> >>> >> >> ran
>> >>> >> >> them, at least one of the feature tests failed. Unfortunately,
>> >>> >> >> it
>> >>> >> >> was a
>> >>> >> >> different one each time. In the last two runs, I got:
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/base_feature_test.py",
>> >>> >> >> line 40, in setUp
>> >>> >> >>     self.before()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 40, in before
>> >>> >> >>     self._connects_to_server()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 144, in _connects_to_server
>> >>> >> >>     self.page.driver.find_element_by_link_text("Create"))\
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 319, in find_element_by_link_text
>> >>> >> >>     return self.find_element(by=By.LINK_TEXT, value=link_text)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >>     'value': value})['value']
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >>     self.error_handler.check_response(response)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >>     raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"link text","selector":"Create"}
>> >>> >> >>   (Session info: chrome=58.0.3029.110)
>> >>> >> >>   (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I checked code from _connects_to_server function which is common
>> >>> >> > in
>> >>> >> > all
>> >>> >> > features test cases. I didn't find anything wrong with this. If
>> >>> >> > there
>> >>> >> > is
>> >>> >> > a
>> >>> >> > bug in this function then all feature test must fail.
>> >>> >> > Let me know if you are getting failure consistently in
>> >>> >> > _connects_to_server
>> >>> >> > function.
>> >>> >>
>> >>> >> I wondered if that one is a race condition. Do we need a short
>> >>> >> delay
>> >>> >> before clicking the Object menu? I have seen this occasionally
>> >>> >> before.
>> >>> >
>> >>> >
>> >>> > OK. In that case let's try putting 1-2 second delay and observer
>> >>> > behaviour.
>> >>> > I'll send separate patch for this tomorrow as this is not related to
>> >>> > on
>> >>> > demand query result feature or its test cases.
>> >>> >
>> >>> >>
>> >>> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 119, in runTest
>> >>> >> >>     self._query_tool_auto_rollback_enabled()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 697, in _query_tool_auto_rollback_enabled
>> >>> >> >>     '//div[contains(@class, "sql-editor-message") and
>> >>> >> >> contains(string(),
>> >>> >> >> "COMMIT")]'
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 295, in find_element_by_xpath
>> >>> >> >>     return self.find_element(by=By.XPATH, value=xpath)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >>     'value': value})['value']
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >>     self.error_handler.check_response(response)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >>     raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"xpath","selector":"//div[contains(@class,
>> >>> >> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
>> >>> >> >>   (Session info: chrome=58.0.3029.110)
>> >>> >> >>   (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I have updated Auto rollback enabled test in this patch.
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Relevant screenshots attached.
>> >>> >> >>
>> >>> >> >> - Can you tidy up the regression output a little please? Instead
>> >>> >> >> of:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ... On demand result set on scrolling...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on grid select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on column select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with verbose...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with costs...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with buffers...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with timing...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit disabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit enabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto rollback enabled...
>> >>> >> >> ERROR
>> >>> >> >> -------
>> >>> >> >>
>> >>> >> >> Something like:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ...
>> >>> >> >> On demand result set on scrolling... OK.
>> >>> >> >> On demand result set on grid select all... OK.
>> >>> >> >> On demand result set on column select all... OK.
>> >>> >> >> Explain query... OK.
>> >>> >> >> Explain query with verbose... OK.
>> >>> >> >> Explain query with costs... OK.
>> >>> >> >> Explain analyze query... OK.
>> >>> >> >> Explain analyze query with buffers... OK.
>> >>> >> >> Explain analyze query with timing... OK.
>> >>> >> >> Auto commit disabled... OK.
>> >>> >> >> Auto commit enabled... OK.
>> >>> >> >> Auto rollback enabled... ERROR
>> >>> >> >> --------
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Thanks!
>> >>> >> >>
>> >>> >> >> --
>> >>> >> >> Dave Page
>> >>> >> >> Blog: http://pgsnake.blogspot.com
>> >>> >> >> Twitter: @pgsnake
>> >>> >> >>
>> >>> >> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> >> The Enterprise PostgreSQL Company
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Dave Page
>> >>> >> Blog: http://pgsnake.blogspot.com
>> >>> >> Twitter: @pgsnake
>> >>> >>
>> >>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> The Enterprise PostgreSQL Company
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Dave Page
>> >>> Blog: http://pgsnake.blogspot.com
>> >>> Twitter: @pgsnake
>> >>>
>> >>> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> The Enterprise PostgreSQL Company
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Dave Page
>> > Blog: http://pgsnake.blogspot.com
>> > Twitter: @pgsnake
>> >
>> > EnterpriseDB UK: http://www.enterprisedb.com
>> > The Enterprise PostgreSQL Company
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

yes i'm working on that only :)

-- 

Harshal Dhumal

Sr. Software Engineer

EnterpriseDB India: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Jun 26, 2017 at 5:22 PM, Dave Page <dpage@pgadmin.org> wrote:

I'm sorry, it needs rebasing again. If you can do it quickly, I'll
make sure it's the next patch I work on in that area.

Thanks.

On Mon, Jun 26, 2017 at 5:16 AM, Harshal Dhumal

<harshal.dhumal@enterprisedb.com> wrote:
> Hi Dave,
>
> Please find updated rebased patch for RM2137
>
> On Fri, Jun 23, 2017 at 9:00 PM, Dave Page <dpage@pgadmin.org> wrote:
>>
>> Hi Harshal,
>>
>> When can we expect an updated version of this patch? I think it's
>> important to get this into the next release.
>>
>> Thanks!
>>
>> On Fri, Jun 16, 2017 at 10:55 AM, Dave Page <dpage@pgadmin.org> wrote:
>> > Hi,
>> >
>> > That's better - the failures are far less random now :-). I got the
>> > following two though, on both PG and EPAS 9.5:
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> > Query tool feature test
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 95, in runTest
>> >     self._query_tool_explain_analyze_buffers()
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> > line 443, in _query_tool_explain_analyze_buffers
>> >     canvas.find_element_by_xpath("//*[contains(string(), 'Shared Read
>> > Blocks')]")
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 260, in find_element_by_xpath
>> >     return self.find_element(by=By.XPATH, value=xpath)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 508, in find_element
>> >     {"using": by, "value": value})['value']
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> >     return self._parent.execute(command, params)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> >     self.error_handler.check_response(response)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> >     raise exception_class(message, screen, stacktrace)
>> > NoSuchElementException: Message: no such element: Unable to locate
>> > element: {"method":"xpath","selector":"//*[contains(string(), 'Shared
>> > Read Blocks')]"}
>> >   (Session info: chrome=58.0.3029.110)
>> >   (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>> >
>> > ======================================================================
>> > ERROR: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 104, in runTest
>> >     self._add_row()
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 255, in _add_row
>> >     self._update_cell(cell_xpath, config_data[str(idx)])
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 164, in _update_cell
>> >     cell_el = self.page.find_by_xpath(xpath)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 122, in find_by_xpath
>> >     return self.wait_for_element(lambda driver:
>> > driver.find_element_by_xpath(xpath))
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 205, in wait_for_element
>> >     return self._wait_for("element to exist", element_if_it_exists)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 255, in _wait_for
>> >     "Timed out waiting for " + waiting_for_message)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/support/wait.py",
>> > line 71, in until
>> >     value = method(self._driver)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/regression/feature_utils/pgadmin_page.py",
>> > line 200, in element_if_it_exists
>> >     if element.is_displayed() and element.is_enabled():
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 358, in is_displayed
>> >     return self._execute(Command.IS_ELEMENT_DISPLAYED)['value']
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py",
>> > line 491, in _execute
>> >     return self._parent.execute(command, params)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> > line 238, in execute
>> >     self.error_handler.check_response(response)
>> >   File
>> > "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> > line 193, in check_response
>> >     raise exception_class(message, screen, stacktrace)
>> > StaleElementReferenceException: Message: stale element reference:
>> > element is not attached to the page document
>> >   (Session info: chrome=58.0.3029.110)
>> >   (Driver info: chromedriver=2.29.461585
>> > (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X 10.12.3
>> > x86_64)
>> >
>
>
> I checked my feature test cases for any database server version/type (PG,
> EPAS) specific failures and I found that test cases are working fine on all
> of them. However I have slightly modified test case to overcome above
> mentioned failures.
>
>>
>> >
>> > On 9.4 everything passes. On 9.6, I get one failure:
>> >
>> > ======================================================================
>> > FAIL: runTest
>> > (pgadmin.feature_tests.view_data_dml_queries.CheckForViewDataTest)
>> > Validate Insert, Update operations in View data with given test data
>> > ----------------------------------------------------------------------
>> > Traceback (most recent call last):
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 105, in runTest
>> >     self._verify_row_data(True)
>> >   File
>> > "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/view_data_dml_queries.py",
>> > line 282, in _verify_row_data
>> >     self.assertEquals(cells[idx], config_data[str(idx)][1])
>> > AssertionError: u'[default]' != u'1'
>> > - [default]
>> > + 1
>> >
>
> I guess Khushboo has fixed this recently.
>
>
>>
>> > Screenshots attached.
>> >
>> > On Fri, Jun 16, 2017 at 7:54 AM, Harshal Dhumal
>> > <harshal.dhumal@enterprisedb.com> wrote:
>> >> Hi Dave,
>> >>
>> >> Please find attached patch where I have added timeout of 2 seconds
>> >> before
>> >> selecting object menu in connect to server function.
>> >>
>> >> Note: Apply this patch on top of previous patch for on demand loading
>> >> feature.
>> >>
>> >>
>> >> --
>> >> Harshal Dhumal
>> >> Sr. Software Engineer
>> >>
>> >> EnterpriseDB India: http://www.enterprisedb.com
>> >> The Enterprise PostgreSQL Company
>> >>
>> >> On Fri, Jun 16, 2017 at 2:25 AM, Dave Page <dpage@pgadmin.org> wrote:
>> >>>
>> >>> Sounds good, thanks.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 9:54 PM, Harshal Dhumal
>> >>> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> > Hi
>> >>> >
>> >>> > On Fri, Jun 16, 2017 at 2:07 AM, Dave Page <dpage@pgadmin.org>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi
>> >>> >>
>> >>> >> On Thu, Jun 15, 2017 at 9:30 PM, Harshal Dhumal
>> >>> >> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> >> > Hi Dave,
>> >>> >> >
>> >>> >> > Please find attached updated patch.
>> >>> >> >
>> >>> >> > On Thu, Jun 15, 2017 at 3:58 PM, Dave Page <dpage@pgadmin.org>
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi
>> >>> >> >>
>> >>> >> >> On Wed, Jun 14, 2017 at 11:36 PM, Harshal Dhumal
>> >>> >> >> <harshal.dhumal@enterprisedb.com> wrote:
>> >>> >> >>>
>> >>> >> >>> Hi Dave,
>> >>> >> >>>
>> >>> >> >>> Please find rebased patch for RM2137.
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> Looking very good. The only issues I see are:
>> >>> >> >>
>> >>> >> >> - The row headers should auto-size such that they can display
>> >>> >> >> the
>> >>> >> >> row
>> >>> >> >> numbers if the last row was displayed. E.g. if there are 12345
>> >>> >> >> rows
>> >>> >> >> in
>> >>> >> >> total, then the row header should be sized to display 5 digits.
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> - The tests are comprehensive, which is awesome. However, every
>> >>> >> >> time
>> >>> >> >> I
>> >>> >> >> ran
>> >>> >> >> them, at least one of the feature tests failed. Unfortunately,
>> >>> >> >> it
>> >>> >> >> was a
>> >>> >> >> different one each time. In the last two runs, I got:
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/regression/feature_utils/base_feature_test.py",
>> >>> >> >> line 40, in setUp
>> >>> >> >>     self.before()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 40, in before
>> >>> >> >>     self._connects_to_server()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 144, in _connects_to_server
>> >>> >> >>     self.page.driver.find_element_by_link_text("Create"))\
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 319, in find_element_by_link_text
>> >>> >> >>     return self.find_element(by=By.LINK_TEXT, value=link_text)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >>     'value': value})['value']
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >>     self.error_handler.check_response(response)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >>     raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"link text","selector":"Create"}
>> >>> >> >>   (Session info: chrome=58.0.3029.110)
>> >>> >> >>   (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I checked code from _connects_to_server function which is common
>> >>> >> > in
>> >>> >> > all
>> >>> >> > features test cases. I didn't find anything wrong with this. If
>> >>> >> > there
>> >>> >> > is
>> >>> >> > a
>> >>> >> > bug in this function then all feature test must fail.
>> >>> >> > Let me know if you are getting failure consistently in
>> >>> >> > _connects_to_server
>> >>> >> > function.
>> >>> >>
>> >>> >> I wondered if that one is a race condition. Do we need a short
>> >>> >> delay
>> >>> >> before clicking the Object menu? I have seen this occasionally
>> >>> >> before.
>> >>> >
>> >>> >
>> >>> > OK. In that case let's try putting 1-2 second delay and observer
>> >>> > behaviour.
>> >>> > I'll send separate patch for this tomorrow as this is not related to
>> >>> > on
>> >>> > demand query result feature or its test cases.
>> >>> >
>> >>> >>
>> >>> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ======================================================================
>> >>> >> >> ERROR: runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> ----------------------------------------------------------------------
>> >>> >> >> Traceback (most recent call last):
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 119, in runTest
>> >>> >> >>     self._query_tool_auto_rollback_enabled()
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/git/pgadmin4/web/pgadmin/feature_tests/query_tool_tests.py",
>> >>> >> >> line 697, in _query_tool_auto_rollback_enabled
>> >>> >> >>     '//div[contains(@class, "sql-editor-message") and
>> >>> >> >> contains(string(),
>> >>> >> >> "COMMIT")]'
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 295, in find_element_by_xpath
>> >>> >> >>     return self.find_element(by=By.XPATH, value=xpath)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 756, in find_element
>> >>> >> >>     'value': value})['value']
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py",
>> >>> >> >> line 238, in execute
>> >>> >> >>     self.error_handler.check_response(response)
>> >>> >> >>   File
>> >>> >> >>
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> "/Users/dpage/.virtualenvs/pgadmin4/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py",
>> >>> >> >> line 193, in check_response
>> >>> >> >>     raise exception_class(message, screen, stacktrace)
>> >>> >> >> NoSuchElementException: Message: no such element: Unable to
>> >>> >> >> locate
>> >>> >> >> element: {"method":"xpath","selector":"//div[contains(@class,
>> >>> >> >> "sql-editor-message") and contains(string(), "COMMIT")]"}
>> >>> >> >>   (Session info: chrome=58.0.3029.110)
>> >>> >> >>   (Driver info: chromedriver=2.29.461585
>> >>> >> >> (0be2cd95f834e9ee7c46bcc7cf405b483f5ae83b),platform=Mac OS X
>> >>> >> >> 10.12.3
>> >>> >> >> x86_64)
>> >>> >> >>
>> >>> >> > I have updated Auto rollback enabled test in this patch.
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Relevant screenshots attached.
>> >>> >> >>
>> >>> >> >> - Can you tidy up the regression output a little please? Instead
>> >>> >> >> of:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ... On demand result set on scrolling...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on grid select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> On demand result set on column select all...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with verbose...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain query with costs...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with buffers...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Explain analyze query with timing...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit disabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto commit enabled...
>> >>> >> >> OK.
>> >>> >> >>
>> >>> >> >> Auto rollback enabled...
>> >>> >> >> ERROR
>> >>> >> >> -------
>> >>> >> >>
>> >>> >> >> Something like:
>> >>> >> >>
>> >>> >> >> -------
>> >>> >> >> runTest
>> >>> >> >> (pgadmin.feature_tests.query_tool_tests.QueryToolFeatureTest)
>> >>> >> >> Query tool feature test ...
>> >>> >> >> On demand result set on scrolling... OK.
>> >>> >> >> On demand result set on grid select all... OK.
>> >>> >> >> On demand result set on column select all... OK.
>> >>> >> >> Explain query... OK.
>> >>> >> >> Explain query with verbose... OK.
>> >>> >> >> Explain query with costs... OK.
>> >>> >> >> Explain analyze query... OK.
>> >>> >> >> Explain analyze query with buffers... OK.
>> >>> >> >> Explain analyze query with timing... OK.
>> >>> >> >> Auto commit disabled... OK.
>> >>> >> >> Auto commit enabled... OK.
>> >>> >> >> Auto rollback enabled... ERROR
>> >>> >> >> --------
>> >>> >> >>
>> >>> >> > Fixed.
>> >>> >> >
>> >>> >> >
>> >>> >> >>
>> >>> >> >> Thanks!
>> >>> >> >>
>> >>> >> >> --
>> >>> >> >> Dave Page
>> >>> >> >> Blog: http://pgsnake.blogspot.com
>> >>> >> >> Twitter: @pgsnake
>> >>> >> >>
>> >>> >> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> >> The Enterprise PostgreSQL Company
>> >>> >> >
>> >>> >> >
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> Dave Page
>> >>> >> Blog: http://pgsnake.blogspot.com
>> >>> >> Twitter: @pgsnake
>> >>> >>
>> >>> >> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> >> The Enterprise PostgreSQL Company
>> >>> >
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Dave Page
>> >>> Blog: http://pgsnake.blogspot.com
>> >>> Twitter: @pgsnake
>> >>>
>> >>> EnterpriseDB UK: http://www.enterprisedb.com
>> >>> The Enterprise PostgreSQL Company
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Dave Page
>> > Blog: http://pgsnake.blogspot.com
>> > Twitter: @pgsnake
>> >
>> > EnterpriseDB UK: http://www.enterprisedb.com
>> > The Enterprise PostgreSQL Company
>>
>>
>>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>
>



--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company