Thread: multi-threaded pgloader needs your tests
Hi, You may remember some thread about data loading performances and multi-threading support in pgloader: http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php The pgloader code to handle this is now ready to get tested, a more structured project could talk about a Release Candidate status. http://pgloader.projects.postgresql.org/dev/TODO.html http://pgloader.projects.postgresql.org/dev/pgloader.1.html#_parallel_loading http://packages.debian.org/pgloader --- experimental has the next version As for the performances benefit of this new version (2.3.0~dev2), all the work could be reduced to zilch because of the python Global Interpreter Lock, which I've been aware of late in the development effort. http://docs.python.org/api/threads.html This documentation states that (a) using generators you're not that concerned, and (b) the global lock still allows for IO and processing at the same time. As pgloader uses generators, I'm still not sure how much a problem this will be. I'd like to have some feedback about the new version, in term of bugs encountered and performance limitations (is pgloader up to what you would expect a multi-threaded loader to be at?) Regards, -- dim
Attachment
On Tue, 2008-02-26 at 13:08 +0100, Dimitri Fontaine wrote: > I'd like to have some feedback about the new version, in term of bugs > encountered and performance limitations (is pgloader up to what you would > expect a multi-threaded loader to be at?) Maybe post to general as well if you don't get any replies here. New feature is very important for us. -- Simon Riggs 2ndQuadrant http://www.2ndQuadrant.com PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk
Hi, Le mardi 26 février 2008, Dimitri Fontaine a écrit : > You may remember some thread about data loading performances and > multi-threading support in pgloader: > http://archives.postgresql.org/pgsql-performance/2008-02/msg00081.php As people here have asked for the new features implemented into pgloader 2.3.0, I'm happy to post here about the availability of the new version! http://pgfoundry.org/projects/pgloader http://pgfoundry.org/forum/forum.php?forum_id=1283 Please consider this as a testbed related to the parallel COPY and pg_restore improvements which have been discussed here and on -hackers, as that's how those new features came to life. Regards, -- dim