Thread: import function's progress report
Hi, all This is a progress report about import function. A screenshot with dialog test mode (/t) attached. Only the file page's code is finished. 1. Target page: Now it is blank. Reserved for standalone version to select target database and table. 2. Source pages: file / Source / ODBC Import from csv format file / other pgsql database / ODBC(for MS-Windows) First, csv file will be supported. Remainders only in plan. 3. Columns page: Set data format ( date & number), space trim and column's order I plan to support express, user can input a pgsql express to treat data. 4. Options/Status/Others page: Only some ideas: Skip empty rows Import rows match regex Don't import rows match regex Commit per (n) rows. Some log options Test insert with (n) rows before insert to DB Generate insert statement to a file. Stop when (n) errors and so on. Limited encoding can be Support, see attached file "encodinglist.txt". ASCII, Japanese EUCJP/Shift_JIS and Chinese GB2312/Big5 had been tested. ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
Attachment
Hi On Wed, Dec 31, 2008 at 7:14 AM, Quan Zongliang <quanzongliang@gmail.com> wrote: > Hi, all > > This is a progress report about import function. > A screenshot with dialog test mode (/t) attached. > Only the file page's code is finished. > > 1. Target page: > Now it is blank. > Reserved for standalone version to select target database and table. OK. Note that the different pages should be on a wxNotebook - don't use buttons to control the UI (we've - OK, I've - made that mistake before). > 2. Source pages: file / Source / ODBC > Import from csv format file / other pgsql database / ODBC(for MS-Windows) > First, csv file will be supported. Remainders only in plan. The different sources should be on a single 'Source' tab. At the top of that tab, use a radio button set to allow the user to select the source type, and have that swap in/out framed sets of controls underneath. For a very basic example of this, look at the Report form which changes the stylesheet options based on whether you select HTML or XML output. > 3. Columns page: > Set data format ( date & number), space trim and column's order > I plan to support express, user can input a pgsql express to treat data. Hmm, pgsql expressions could be slow as they can only be evaluated in insert statements, which is really not good for such a tool. I would suggest: - Consider embedding Python (which is on the TODO anyway) and allowing transforms to be scripted on the client. - Convert the import data to a COPY compatible format so it can be fed directly to the server without using INSERT which is *much* slower. > 4. Options/Status/Others page: > Only some ideas: > Skip empty rows > Import rows match regex > Don't import rows match regex > Commit per (n) rows. > Some log options > Test insert with (n) rows before insert to DB > Generate insert statement to a file. > Stop when (n) errors > and so on. Sounds good :-) > Limited encoding can be Support, see attached file "encodinglist.txt". > ASCII, Japanese EUCJP/Shift_JIS and Chinese GB2312/Big5 had been tested. Cool :-) -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Dave Page wrote: > Hi > > On Wed, Dec 31, 2008 at 7:14 AM, Quan Zongliang <quanzongliang@gmail.com> wrote: >> Hi, all >> 3. Columns page: >> Set data format ( date & number), space trim and column's order >> I plan to support express, user can input a pgsql express to treat data. > > Hmm, pgsql expressions could be slow as they can only be evaluated in > insert statements, which is really not good for such a tool. I would > suggest: > > - Consider embedding Python (which is on the TODO anyway) and allowing > transforms to be scripted on the client. > - Convert the import data to a COPY compatible format so it can be fed > directly to the server without using INSERT which is *much* slower. Agreed, and a feature I would very much like to see. FYI, I recently wrote a small tool that did *just* this step (the conversion while loading). When running both the client and the server on the same machine, it still came out more than 3x faster if I did the simple transforms in Python and maintained the COPY format rather than switching to INSERT format. Do you plan to support loading multiple tables in one batch (transaction)? If not initially, it would be great if you could plan for the ability to do so on in the future - that is, don't make assumptions that we will not want it. Because that would be a great feature. //Magnus
On Wed, Dec 31, 2008 at 11:18 AM, Quan Zongliang <quanzongliang@gmail.com> wrote: >> OK. Note that the different pages should be on a wxNotebook - don't >> use buttons to control the UI (we've - OK, I've - made that mistake >> before). > > > I use wxToolbook and writed a XML handler. Because I can not find method > to hide or disable tabs with wxNotebook. > The UI is controlled by "Back" and "Next" button. Only current page > enbled. This is easy to code (I am lazy -.-). It can be modified if need. If you want to use Next/Previous buttons, then you should certainly be using a wxWizard. They're actually pretty easy to code. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
> OK. Note that the different pages should be on a wxNotebook - don't > use buttons to control the UI (we've - OK, I've - made that mistake > before). I use wxToolbook and writed a XML handler. Because I can not find method to hide or disable tabs with wxNotebook. The UI is controlled by "Back" and "Next" button. Only current page enbled. This is easy to code (I am lazy -.-). It can be modified if need. > > The different sources should be on a single 'Source' tab. At the top > of that tab, use a radio button set to allow the user to select the Now, When open the import dialog, only one source page can be displayed. Like the attachment. > - Consider embedding Python (which is on the TODO anyway) and allowing > transforms to be scripted on the client. I know nothing about Python. Left it later. :) > - Convert the import data to a COPY compatible format so it can be fed > directly to the server without using INSERT which is *much* slower. OK, good idea. ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
Attachment
Dave Page wrote: > On Wed, Dec 31, 2008 at 11:18 AM, Quan Zongliang > <quanzongliang@gmail.com> wrote: >>> OK. Note that the different pages should be on a wxNotebook - don't >>> use buttons to control the UI (we've - OK, I've - made that mistake >>> before). >> >> I use wxToolbook and writed a XML handler. Because I can not find method >> to hide or disable tabs with wxNotebook. >> The UI is controlled by "Back" and "Next" button. Only current page >> enbled. This is easy to code (I am lazy -.-). It can be modified if need. > > If you want to use Next/Previous buttons, then you should certainly be > using a wxWizard. They're actually pretty easy to code. Hmm. It being a wizard seems a bit strange to me though - given what's on the tabs, a regular tabbed dialog seems a lot more logical? (even if it's harder to code..) //Magnus
Quan Zongliang wrote: > enbled. This is easy to code (I am lazy -.-). It can be modified if need. >> The different sources should be on a single 'Source' tab. At the top >> of that tab, use a radio button set to allow the user to select the > > Now, When open the import dialog, only one source page can be displayed. > Like the attachment. > > >> - Consider embedding Python (which is on the TODO anyway) and allowing >> transforms to be scripted on the client. > I know nothing about Python. Left it later. :) Heh. Well, please consider leaving a good "hook point" for such an integration to go in, and I'll see if I can find the time to work on that. //Magnus
> Do you plan to support loading multiple tables in one batch > (transaction)? If not initially, it would be great if you could plan for > the ability to do so on in the future - that is, don't make assumptions > that we will not want it. Because that would be a great feature. No plan, but it is a good idea. I will think about it. ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
> If you want to use Next/Previous buttons, then you should certainly be > using a wxWizard. They're actually pretty easy to code. Ok, I will look to it. And ... the work will go on with wxToolbook, as I know nothing about wxWizard now. Maybe the users would like to use style like wxNotebook? It can be free to select any option page. And To all: Happy new year! See you in 2009. ^.^ ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
On Wed, Dec 31, 2008 at 11:50 AM, Quan Zongliang <quanzongliang@gmail.com> wrote: >> If you want to use Next/Previous buttons, then you should certainly be >> using a wxWizard. They're actually pretty easy to code. > Ok, I will look to it. > And ... the work will go on with wxToolbook, as I know nothing about > wxWizard now. > Maybe the users would like to use style like wxNotebook? > It can be free to select any option page. No, we don't make UI design optional - we use the appropriate controls for the task and leave it at that. > And > To all: > Happy new year! > See you in 2009. And to you! -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
The work restarted. > No, we don't make UI design optional - we use the appropriate controls > for the task and leave it at that. I had read wxWizard's document. The button's label and status can't be controlled. But I think we need <start> <stop> or <pause> buttons and labels or more controller to control the data import progress. One choice: We can use wxWizardPage to contain this contrllers, as it derived from wxPanel. But this may make UI look strange. Two: Rewrite the wxWizard class, add some Get and Set method to control more elements. Which? ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
> You can use code such as: > > FindWindow(wxID_FORWARD)->Disable(); > > Right - but those controls can be incorporated onto a panel that > provides additional user feedback such as a progress bar - and just > disable the Next/Finish button until the import has completed. > > Please bear in mind that there are likely to be more wizards in the > future (for example, to aid replication setup). Whatever you do here > ... Ok, got. ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn
On Sun, Jan 4, 2009 at 7:46 AM, Quan Zongliang <quanzongliang@gmail.com> wrote: > > The work restarted. > >> No, we don't make UI design optional - we use the appropriate controls >> for the task and leave it at that. > I had read wxWizard's document. The button's label and status can't be > controlled. You can use code such as: FindWindow(wxID_FORWARD)->Disable(); > But I think we need <start> <stop> or <pause> buttons and > labels or more controller to control the data import progress. Right - but those controls can be incorporated onto a panel that provides additional user feedback such as a progress bar - and just disable the Next/Finish button until the import has completed. > One choice: > We can use wxWizardPage to contain this contrllers, as it derived from > wxPanel. But this may make UI look strange. I don't think so, if it's designed nicely. > Two: > Rewrite the wxWizard class, add some Get and Set method to control more > elements. There's no reason we cannot extend the class anyway, to provide cleaner access to the buttons etc. Please bear in mind that there are likely to be more wizards in the future (for example, to aid replication setup). Whatever you do here also defines the style of those future wizards as we don't want to have differing look and feel between them. Therefore, I would suggest you need to keep things as simple as possible and only deviate form the wxWidgets way of doing things where absolutely necessary. -- Dave Page EnterpriseDB UK: http://www.enterprisedb.com
Magnus Hagander escreveu: > Quan Zongliang wrote: > >>> - Consider embedding Python (which is on the TODO anyway) and allowing >>> transforms to be scripted on the client. >>> >> I know nothing about Python. Left it later. :) >> > > Heh. > > Well, please consider leaving a good "hook point" for such an > integration to go in, and I'll see if I can find the time to work on that Are somebody working on an embbed python in pgadmin? -- Dickson S. Guedes Administrador de Banco de Dados Confesol - Projeto Colmeia Florianopolis, SC, Brasil (48) 3322-1185, ramal: 26
> Are somebody working on an embbed python in pgadmin? TODO Item: - Python scripting engine? No one doing it. ----------------------------------------------- Quan Zongliang quanzongliang@gmail.com CIT Japan: http://www.cit.co.jp CIT China: http://www.citbj.com.cn