Re: [pgAdmin4][patch][runtime]: RM#2829, RM#2491 - pgAdmin4 crasheswhile saving CSV data from Query tool - Mailing list pgadmin-hackers

Hi Dave,

I have tried for 350K rows with query given by you but not able to reproduce the hang issue.

I have put some debug statement in Qt Signal's handler. Can you please apply this debug patch and provide me console output if possible ?

Thanks,

Neel Patel

On Fri, Nov 17, 2017 at 3:13 PM, Neel Patel <neel.patel@enterprisedb.com> wrote:

Hi Dave,

On Fri, Nov 17, 2017 at 3:09 PM, Dave Page <dpage@pgadmin.org> wrote:

On Fri, Nov 17, 2017 at 9:36 AM, Neel Patel <neel.patel@enterprisedb.com> wrote:

Hi Dave,

On Fri, Nov 17, 2017 at 2:42 PM, Dave Page <dpage@pgadmin.org> wrote:

On Fri, Nov 17, 2017 at 8:45 AM, Neel Patel <neel.patel@enterprisedb.com> wrote:

Hi Dave,

On Thu, Nov 16, 2017 at 8:13 PM, Dave Page <dpage@pgadmin.org> wrote:

Hi

On Thu, Nov 16, 2017 at 1:47 PM, Neel Patel <neel.patel@enterprisedb.com> wrote:

Hi,

Adding information.

With this patch, RM#2715 should also be resolved.

Thanks,

Neel Patel

On Thu, Nov 16, 2017 at 7:01 PM, Neel Patel <neel.patel@enterprisedb.com> wrote:

Hi,

I am able to reproduce the crash while downloading and save data in CSV file from query tool. 

Please find attached updated patch with below changes after reading Qt documentation.

• Added new signal "readyRead". As per the Qt documentation, this signal will be emitted when data is ready from IO channel for large amount of data transfer between server and client.
• Ready read and DownloadInProgress signal is very quick in call so we should not do large operation inside that slot because for downloading big data it may possible of frequent call of those signals which may cause the crash or missing data to write inside the file so removed unnecessary logic from that slot.
• Fixed the crash while opening IODevice with NULL handle.

With above changes, I have tested with same data as earlier and it is working as expected without crashing the application.

Do review it and let me know for comments.

My first test was on Mac using Qt 5.8 with webkit, and when I attempted to download the results from a 300K row query, it basically hung showing the Downloading File progress indicator. It let me cancel it and carried on working, but hung again the next time I tried the CSV download. Each time it seems to download some data - from the same query I've seen 2.8MB, 5.1MB and 1.5MB.

We are using "downloadProgress"  signal to update the progress bar and here we are getting "readData" as bytes but "totalData" is unknown so we received "totalData" as -1 because Qt doesn't know the total size of the file we are getting from server but when we open "pgAdmin4" website in another link and try to download some binary or source at that time we are getting "totalData" as known size and we are displaying the progress bar correctly.

Here, Qt sending the signal "downloadProgress" with unknown totalData size as -1 so we update the progress bar with 100% so from user point of view it looks like hang but it is downloading and writing to file at back ground. if we apply the same patch in windows then it display running progress bar so user can know something happening at back end side.

I think it is same as browser - if we download big size file from server then it keep on downloading the file without displaying total size. What should we do here ? Should we create one custom progress bar dialog with moving cursor or something other so that user can know that activity is doing at backend ? Thoughts ?

It is misleading, but it *is* also hanging. 10+ minutes to download a few megs of data from a server on the same machine is not just a misleading progress indicator :-) 

Can you please share some table definition with no of rows with ~table size ? I have tried but not able to reproduce the issue. 

It was simply (on PG 9.4):

SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid

UNION ALL

SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid

UNION ALL

SELECT * FROM pg_class c, pg_attribute a WHERE c.oid = a.attrelid

UNION ALL

...

Unioned enough times to get a shade under 300K rows.

Thanks. I will check and keep you updated.

--

Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company