Thread: Testing with concurrent sessions
There has been periodic discussion here about allowing psql to deal with multiple sessions, or possibly creating another tool to allow this sort of test. Is anyone working on this? It's very soon going to be critical that I be able to test particular interleavings of statements in particular concurrent transaction sets to be able to make meaningful progress on the serializable transaction work. It would be wonderful if some of these scripts could be integrated into the PostgreSQL 'make check' scripts, although that's not an absolute requirement. I'm not really concerned about performance tests for a while, just testing the behavior of particular interleavings of statements in multiple sessions. If psql isn't expected to support that soon, any suggestions? Is pgTAP suited to this? -Kevin
On Jan 1, 2010, at 6:01 PM, Kevin Grittner wrote: > It's very soon going to be critical that I be able to test particular > interleavings of statements in particular concurrent transaction sets > to be able to make meaningful progress on the serializable > transaction work. It would be wonderful if some of these scripts > could be integrated into the PostgreSQL 'make check' scripts, > although that's not an absolute requirement. I'm not really > concerned about performance tests for a while, just testing the > behavior of particular interleavings of statements in multiple > sessions. If psql isn't expected to support that soon, any > suggestions? Is pgTAP suited to this? We've discussed it a bit in the past with regard to testing replication and such. I think the consensus was, failing supportfor concurrent sessions in psql, to use a Perl script to control multiple psql sessions and perhaps use Test::Moreto do the testing. Although pgTAP might make sense, too, if the tests ought to run in the database. Best, David
"David E. Wheeler" <david@kineticode.com> wrote: > I think the consensus was, failing support for concurrent sessions > in psql, to use a Perl script to control multiple psql sessions > and perhaps use Test::More to do the testing. Are there any examples of that? While I can hack my way through regular expressions when I need them, perl as a language is something I don't know at all; with an example I might be able to come up to speed quickly, though. > Although pgTAP might make sense, too, if the > tests ought to run in the database. I need to run statements against a database; I don't particularly need any special features of psql for this. Can anyone confirm that pgTAP can let you interleave specific statements against specific connections in a specific sequence? (The answer to that didn't leap out at me in a quick scan of the docs.) -Kevin
On mån, 2010-01-04 at 17:10 -0600, Kevin Grittner wrote: > "David E. Wheeler" <david@kineticode.com> wrote: > > > I think the consensus was, failing support for concurrent sessions > > in psql, to use a Perl script to control multiple psql sessions > > and perhaps use Test::More to do the testing. > > Are there any examples of that? While I can hack my way through > regular expressions when I need them, perl as a language is > something I don't know at all; with an example I might be able to > come up to speed quickly, though. If you're not deep into Perl, perhaps ignore the Test::More comment for the moment and just use DBI to connect to several database sessions, execute your queries and check if the results are what you want. Once you have gotten somewhere with that, wrapping a test harness around it is something others will be able to help with. > > Although pgTAP might make sense, too, if the > > tests ought to run in the database. > > I need to run statements against a database; I don't particularly > need any special features of psql for this. Can anyone confirm that > pgTAP can let you interleave specific statements against specific > connections in a specific sequence? (The answer to that didn't leap > out at me in a quick scan of the docs.) pgTAP isn't really going to help you here, as it runs with *one* database session, and its main functionality is to format the result of SQL functions into TAP output, which is not very much like what you ought to be doing.
On Jan 4, 2010, at 3:29 PM, Peter Eisentraut wrote: > If you're not deep into Perl, perhaps ignore the Test::More comment for > the moment and just use DBI to connect to several database sessions, > execute your queries and check if the results are what you want. Once > you have gotten somewhere with that, wrapping a test harness around it > is something others will be able to help with. Last I heard, Andrew was willing to require Test::More for testing, so that a Perl script could handle multiple psql connections(perhaps forked) and output test results based on them. But he wasn't as interested in requiring DBI and DBD::Pg,neither of which are in the Perl core and are more of a PITA to install (not huge, but the barrier might as wellstay low). > pgTAP isn't really going to help you here, as it runs with *one* > database session, and its main functionality is to format the result of > SQL functions into TAP output, which is not very much like what you > ought to be doing. Right, exactly. Best, David
Hi, Kevin Grittner wrote: > It's very soon going to be critical that I be able to test particular > interleavings of statements in particular concurrent transaction sets > to be able to make meaningful progress on the serializable > transaction work. I've something in place for Postgres-R, as I also need to test concurrent transactions there. It's based on python/twisted and is able to start multiple Postgres instances (as required for testing replication) and query them concurrently (as you seem to need as well). It uses an asynchronous event loop (from twisted) and basically controls processes, issues queries and checks results and ordering constraints (e.g. transaction X must commit and return a result before transaction Y). I'm still under the impression that this testing framework needs cleanup. However, others already showed interest as well... Regards Markus Wanner
Markus Wanner <markus@bluegap.ch> wrote: > Kevin Grittner wrote: >> It's very soon going to be critical that I be able to test >> particular interleavings of statements in particular concurrent >> transaction sets to be able to make meaningful progress on the >> serializable transaction work. > > I've something in place for Postgres-R, as I also need to test > concurrent transactions there. It's based on python/twisted and is > able to start multiple Postgres instances (as required for testing > replication) and query them concurrently (as you seem to need as > well). It uses an asynchronous event loop (from twisted) and > basically controls processes, issues queries and checks results > and ordering constraints (e.g. transaction X must commit and > return a result before transaction Y). Where would I find this (and any related documentation)? -Kevin
Hi, Kevin Grittner wrote: > Where would I find this (and any related documentation)? Sorry, if that didn't get clear. I'm trying to put together something I can release real soon now (tm). I'll keep you informed. Regards Markus Wanner
"David E. Wheeler" <david@kineticode.com> wrote: > Last I heard, Andrew was willing to require Test::More for > testing, so that a Perl script could handle multiple psql > connections (perhaps forked) and output test results based on > them. But he wasn't as interested in requiring DBI and DBD::Pg, > neither of which are in the Perl core and are more of a PITA to > install (not huge, but the barrier might as well stay low). OK, I've gotten familiar with Perl as a programming language and tinkered with Test::More. What's not clear to me yet is what would be considered good technique for launching several psql sessions from that environment, interleaving commands to each of them, and checking results from each of them as the test plan progresses. Any code snippets or URLs to help me understand that are welcome. (It seems clear enough with DBI, but I'm trying to avoid that per the above.) -Kevin
On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote: > "David E. Wheeler" <david@kineticode.com> wrote: > > > Last I heard, Andrew was willing to require Test::More for > > testing, so that a Perl script could handle multiple psql > > connections (perhaps forked) and output test results based on > > them. But he wasn't as interested in requiring DBI and DBD::Pg, > > neither of which are in the Perl core and are more of a PITA to > > install (not huge, but the barrier might as well stay low). > > OK, I've gotten familiar with Perl as a programming language and > tinkered with Test::More. What's not clear to me yet is what would > be considered good technique for launching several psql sessions > from that environment, interleaving commands to each of them, and > checking results from each of them as the test plan progresses. Any > code snippets or URLs to help me understand that are welcome. (It > seems clear enough with DBI, but I'm trying to avoid that per the > above.) Then I don't see much of a point in using Perl. You might as well fire up a few psqls from a shell script.
On Jan 6, 2010, at 1:52 PM, Kevin Grittner wrote: >> Last I heard, Andrew was willing to require Test::More for >> testing, so that a Perl script could handle multiple psql >> connections (perhaps forked) and output test results based on >> them. But he wasn't as interested in requiring DBI and DBD::Pg, >> neither of which are in the Perl core and are more of a PITA to >> install (not huge, but the barrier might as well stay low). > > OK, I've gotten familiar with Perl as a programming language and > tinkered with Test::More. What's not clear to me yet is what would > be considered good technique for launching several psql sessions > from that environment, interleaving commands to each of them, and > checking results from each of them as the test plan progresses. Any > code snippets or URLs to help me understand that are welcome. (It > seems clear enough with DBI, but I'm trying to avoid that per the > above.) Probably the simplest way is to use the core IPC::Open3 module: http://search.cpan.org/perldoc?IPC::Open3 IPC::Run might be easier to use if it's available, but it's not in Perl core, so YMMV. Really it's up to andrew what moduleshe requires test servers to have. Best, David
On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote: > Then I don't see much of a point in using Perl. You might as well fire > up a few psqls from a shell script If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it? Best, David
On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: > On ons, 2010-01-06 at 15:52 -0600, Kevin Grittner wrote: >> "David E. Wheeler"<david@kineticode.com> wrote: >> >>> Last I heard, Andrew was willing to require Test::More for >>> testing, so that a Perl script could handle multiple psql >>> connections (perhaps forked) and output test results based on >>> them. But he wasn't as interested in requiring DBI and DBD::Pg, >>> neither of which are in the Perl core and are more of a PITA to >>> install (not huge, but the barrier might as well stay low). >> >> OK, I've gotten familiar with Perl as a programming language and >> tinkered with Test::More. What's not clear to me yet is what would >> be considered good technique for launching several psql sessions >> from that environment, interleaving commands to each of them, and >> checking results from each of them as the test plan progresses. Any >> code snippets or URLs to help me understand that are welcome. (It >> seems clear enough with DBI, but I'm trying to avoid that per the >> above.) > > Then I don't see much of a point in using Perl. You might as well fire > up a few psqls from a shell script. I don't see how that would work, but I might have misunderstood what we're reaching for here. What I think would be most useful would be to interleave statements between transactions, not just randomly fire psql sessions and hope for race conditions. Regards, Marko Tiikkaja
Marko Tiikkaja <marko.tiikkaja@cs.helsinki.fi> wrote: > On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: >> Then I don't see much of a point in using Perl. You might as >> well fire up a few psqls from a shell script. > > I don't see how that would work, but I might have misunderstood > what we're reaching for here. What I think would be most useful > would be to interleave statements between transactions, not just > randomly fire psql sessions and hope for race conditions. Yeah, I want to test specific interleavings of statements on concurrent connections. There may *also* be some tests which throw a lot at the server concurrently in a more random fashion, but it is important to be able to have some very controlled tests where we don't count on randomly creating the desired conflicts. It would be valuable to be able to include some of these tests with controlled and predicatable statement interleavings in the "make check" tests. -Kevin
Marko Tiikkaja wrote: > On 2010-01-07 00:08 +0200, Peter Eisentraut wrote: > >Then I don't see much of a point in using Perl. You might as well fire > >up a few psqls from a shell script. > > I don't see how that would work, but I might have misunderstood what > we're reaching for here. What I think would be most useful would be > to interleave statements between transactions, not just randomly > fire psql sessions and hope for race conditions. Open a few psql with -f pointing to a pipe, and from the shell write into the pipe? I don't think it's straightforward, but it should be possible. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Alvaro Herrera <alvherre@commandprompt.com> wrote: > Open a few psql with -f pointing to a pipe, and from the shell > write into the pipe? I don't think it's straightforward, but it > should be possible. I'll play with it and see what I can do. Thanks, -Kevin
On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: > "David E. Wheeler" <david@kineticode.com> wrote: > >> Last I heard, Andrew was willing to require Test::More for >> testing, so that a Perl script could handle multiple psql >> connections (perhaps forked) and output test results based on >> them. But he wasn't as interested in requiring DBI and DBD::Pg, >> neither of which are in the Perl core and are more of a PITA to >> install (not huge, but the barrier might as well stay low). > > OK, I've gotten familiar with Perl as a programming language and > tinkered with Test::More. What's not clear to me yet is what would > be considered good technique for launching several psql sessions > from that environment, interleaving commands to each of them, and > checking results from each of them as the test plan progresses. Any > code snippets or URLs to help me understand that are welcome. (It > seems clear enough with DBI, but I'm trying to avoid that per the > above.) Doing this without DBI is going to be ten times harder than doing it with DBI. Are we really sure that's not a viable option? ...Robert
Robert Haas wrote: > On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner > <Kevin.Grittner@wicourts.gov> wrote: > >> "David E. Wheeler" <david@kineticode.com> wrote: >> >> >>> Last I heard, Andrew was willing to require Test::More for >>> testing, so that a Perl script could handle multiple psql >>> connections (perhaps forked) and output test results based on >>> them. But he wasn't as interested in requiring DBI and DBD::Pg, >>> neither of which are in the Perl core and are more of a PITA to >>> install (not huge, but the barrier might as well stay low). >>> >> OK, I've gotten familiar with Perl as a programming language and >> tinkered with Test::More. What's not clear to me yet is what would >> be considered good technique for launching several psql sessions >> from that environment, interleaving commands to each of them, and >> checking results from each of them as the test plan progresses. Any >> code snippets or URLs to help me understand that are welcome. (It >> seems clear enough with DBI, but I'm trying to avoid that per the >> above.) >> > > Doing this without DBI is going to be ten times harder than doing it > with DBI. Are we really sure that's not a viable option? > > > In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that it should do what you would do yourself by hand. And adding DBI as a requirement for running a buildfarm member would be a significant extra barrier to entry, ISTM. (I am very fond of DBI, and use it frequently, BTW) I'm persuadable on most things, but this one would take a bit of doing. A parallel psql seems to me a better way to go. We talked about that a while ago, but I don't recall what happened to it. cheers andrew
On Wed, Jan 6, 2010 at 8:40 PM, Andrew Dunstan <andrew@dunslane.net> wrote: >> Doing this without DBI is going to be ten times harder than doing it >> with DBI. Are we really sure that's not a viable option? > > In the buildfarm? Yes, I think so. The philosophy of the buildfarm is that > it should do what you would do yourself by hand. It just seems crazy to me to try to test anything without proper language bindings. Opening a psql session and parsing the results seems extraordinarily painful. I wonder if it would make sense write a small wrapper program that uses libpq and dumps out the results in a format that is easy for Perl to parse. Another idea would be to make a set of Perl libpq bindings that is simpler than DBD::Pg and don't go through DBI. If we put those in the main source tree (perhaps as a contrib module) they would be available wherever we need them. > A parallel psql seems to me a better way to go. We talked about that a while > ago, but I don't recall what happened to it. That seems like a dead-end to me. It's hard for me to imagine it's ever going to be more than a toy. ...Robert
Robert Haas <robertmhaas@gmail.com> writes:
> It just seems crazy to me to try to test anything without proper
> language bindings. Opening a psql session and parsing the results
> seems extraordinarily painful. I wonder if it would make sense write
> a small wrapper program that uses libpq and dumps out the results in a
> format that is easy for Perl to parse.
> Another idea would be to make a set of Perl libpq bindings that is
> simpler than DBD::Pg and don't go through DBI. If we put those in the
> main source tree (perhaps as a contrib module) they would be available
> wherever we need them.
We have not yet fully accepted the notion that you must have Perl to
build (and, in fact, I am right now trying to verify that you don't).
I don't think that requiring Perl to test is going to fly.
>> A parallel psql seems to me a better way to go. We talked about that a while
>> ago, but I don't recall what happened to it.
> That seems like a dead-end to me. It's hard for me to imagine it's
> ever going to be more than a toy.
Well, the argument there is that it might be useful for actual use,
not only testing.
regards, tom lane
Andrew Dunstan wrote: > Robert Haas wrote: >> Doing this without DBI is going to be ten times harder than doing >> it with DBI. Are we really sure that's not a viable option? > In the buildfarm? Yes, I think so. The philosophy of the buildfarm > is that it should do what you would do yourself by hand. > > And adding DBI as a requirement for running a buildfarm member > would be a significant extra barrier to entry, ISTM. (I am very > fond of DBI, and use it frequently, BTW) > > I'm persuadable on most things, but this one would take a bit of > doing. As far as I've been able to determine so far, to call psql in a relatively portable way would require something like this: http://perldoc.perl.org/perlfork.html Is that really better than DBI? Don't we need some way to routinely test multi-session issues? Other ideas? -Kevin
Tom Lane wrote: > We have not yet fully accepted the notion that you must have Perl > to build (and, in fact, I am right now trying to verify that you > don't). I don't think that requiring Perl to test is going to fly. Well, if that's the consensus, I have to choose between trying to implement multi-session psql and using testing which can't carry over to long-term regular use. Are we anywhere close to an agreement on what the multi-session psql implementation would look like? (If not I can put something forward.) -Kevin
On Wed, Jan 6, 2010 at 9:26 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Robert Haas <robertmhaas@gmail.com> writes: >> It just seems crazy to me to try to test anything without proper >> language bindings. Opening a psql session and parsing the results >> seems extraordinarily painful. I wonder if it would make sense write >> a small wrapper program that uses libpq and dumps out the results in a >> format that is easy for Perl to parse. > >> Another idea would be to make a set of Perl libpq bindings that is >> simpler than DBD::Pg and don't go through DBI. If we put those in the >> main source tree (perhaps as a contrib module) they would be available >> wherever we need them. > > We have not yet fully accepted the notion that you must have Perl to > build (and, in fact, I am right now trying to verify that you don't). > I don't think that requiring Perl to test is going to fly. I suppose that depends on the context. I'm not exactly sure what Kevin's goal is here. For basic regression tests, yeah, we'd probably like to keep that Perl-free. For more complex testing, I think using Perl makes sense. Or to put the shoe on the other foot, if we DON'T allow the use of Perl for more complex testing, then we're probably not going to have any more complex tests. If we use a hypothetical concurrent psql implementation to run the tests, how will we analyze the results? It's no secret that the current regression tests are fairly limited, in part because the only thing we can do with them is diff the output against one or two "known good" results. ...Robert
Robert Haas wrote: > I'm not exactly sure what Kevin's goal is here. I think it would be insane to try to do something like serializable isolation mode without having regression tests. I would want more tests than could reasonably go into the 'make check' suite to support development, but it would be very good to have some go in there. > For basic regression tests, yeah, we'd probably like to keep that > Perl-free. OK. Is parallel psql the only reasonable option? > For more complex testing, I think using Perl makes sense. Or to put > the shoe on the other foot, if we DON'T allow the use of Perl for > more complex testing, then we're probably not going to have any > more complex tests. Do you envision some test suite committed to CVS beyond the 'make check' tests, for "on demand" testing at a more rigorous level? Am I missing something that's already there? -Kevin
Kevin Grittner escribió: > Well, if that's the consensus, I have to choose between trying to > implement multi-session psql and using testing which can't carry over > to long-term regular use. Are we anywhere close to an agreement on > what the multi-session psql implementation would look like? (If not > I can put something forward.) See http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us and followups. -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Wed, Jan 6, 2010 at 10:00 PM, Kevin Grittner <Kevin.Grittner@wicourts.gov> wrote: >> For basic regression tests, yeah, we'd probably like to keep that >> Perl-free. > > OK. Is parallel psql the only reasonable option? It seems so, assuming you're willing to concede that it is a reasonable option in the first place. >> For more complex testing, I think using Perl makes sense. Or to put >> the shoe on the other foot, if we DON'T allow the use of Perl for >> more complex testing, then we're probably not going to have any >> more complex tests. > > Do you envision some test suite committed to CVS beyond the 'make > check' tests, for "on demand" testing at a more rigorous level? > Am I missing something that's already there? Personally, I tend to think that to test this well you are going to need a test suite written in a scripting language. Whether or not that gets committed to CVS is a political question, but I would be in favor of it (assuming it's good, of course). Maybe you will find that you can do it all with concurrent psql, but (1) I'm not convinced and (2) if that's your plan, does that mean you're going to do nothing until someone implements concurrent psql? ...Robert
Robert Haas wrote: > It just seems crazy to me to try to test anything without proper > language bindings. Opening a psql session and parsing the results > seems extraordinarily painful. I've written a Python based program that spawns a captive psql and talks to it--twice for different people--that ultimately uses the same sort of open3() spawning David mentioned is available via IPC::Open3. You can throw together a prototype that works well enough for some purposes in a couple of hours. I don't know that it would ever reach really robust though. The main problem with that whole approach is that you have to be extremely careful in how you deal with the situation where the captive program is spewing an unknown amount of information back at you. How do you know when it's done? Easy for the child and its master to deadlock if you're not careful. In the past I worked around that issue by just waiting for the process to end and then returning everything it had written until that time. I don't know that this would be flexible enough for what's needed for concurrent testing, where people are probably going to want more of a "send a command, get some lines back again" approach that keeps the session open. If I thought a captive psql would work well in this context I'd have written a prototype already. I'm not sure if it's actually possible to do this well enough to meet expectations. Parsing psql output is completely viable for trivial purposes though, and if the requirements were constrained enough it might work well enough for simple concurrent testing. While both concurrent psql and the libpq shim you suggested would take more work, I feel a bit more confident those would result in something that really worked as expected on every platform when finished. -- Greg Smith 2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support greg@2ndQuadrant.com www.2ndQuadrant.com
On ons, 2010-01-06 at 14:10 -0800, David E. Wheeler wrote: > On Jan 6, 2010, at 2:08 PM, Peter Eisentraut wrote: > > > Then I don't see much of a point in using Perl. You might as well fire > > up a few psqls from a shell script > > If you're more comfortable with shell, then yes. Although then it won't run on Windows, will it? Well, I'm not saying that this is necessarily the better alternative. But you might have to compromise somewhere. Otherwise this project will take two years to complete and will be an unmaintainable mess (see pg_regress).
On ons, 2010-01-06 at 20:49 -0500, Robert Haas wrote: > Another idea would be to make a set of Perl libpq bindings that is > simpler than DBD::Pg and don't go through DBI. If we put those in the > main source tree (perhaps as a contrib module) they would be available > wherever we need them. http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/perl5/?hideattic=0
On 7/01/2010 9:15 AM, Robert Haas wrote: > On Wed, Jan 6, 2010 at 4:52 PM, Kevin Grittner > <Kevin.Grittner@wicourts.gov> wrote: >> "David E. Wheeler"<david@kineticode.com> wrote: >> >>> Last I heard, Andrew was willing to require Test::More for >>> testing, so that a Perl script could handle multiple psql >>> connections (perhaps forked) and output test results based on >>> them. But he wasn't as interested in requiring DBI and DBD::Pg, >>> neither of which are in the Perl core and are more of a PITA to >>> install (not huge, but the barrier might as well stay low). >> >> OK, I've gotten familiar with Perl as a programming language and >> tinkered with Test::More. What's not clear to me yet is what would >> be considered good technique for launching several psql sessions >> from that environment, interleaving commands to each of them, and >> checking results from each of them as the test plan progresses. Any >> code snippets or URLs to help me understand that are welcome. (It >> seems clear enough with DBI, but I'm trying to avoid that per the >> above.) > > Doing this without DBI is going to be ten times harder than doing it > with DBI. Are we really sure that's not a viable option? At this point, I'm personally wondering if it's worth putting together a simple (ish) C program that reads a file describing command interleavings on n connections. It fires up one thread per connection required, then begins queuing commands up for the threads to execute in per-thread fifo queues. The input file may contain synchronization points where two or more explicitly specified threads (or just all threads) must finish all their queued work before they may be given more. Yes, it requires wrangling low-level threading ( pthreads, or the practically equivalent for simple purposes but differently spelled win32 threading ) so it's not going to be beautiful. But it'd permit a declarative form for tests and a single, probably fairly maintainable, test runner. I reluctantly suspect that XML would be a good way to describe the tests - first a block declaring your connections and their conn strings, then a sequence of statements (each of which is associated with a named connection) and synchronization points. Though, come to think of it, a custom plaintext format would be pretty trivial too. CONN conn1: dbname=regress, user=regress CONN conn2: dbname=regress, user=regress STMT conn1: SELECT blah blah; STMT conn2: UPDATE blah blah; SYNC conn1, conn2 etc. Or alternately one-file-per-connection (which would be handy if one connection has *lots* of commands and others only occasional ones) - the only trouble there being how to conveniently specify synchronization points. Anyway: If Java were acceptable I'd put one together now - but somehow I don't think requiring Java would be popular if Perl is an issue ;-) My C/pthreads is more than a little bit rusty (ie: practially nonexistent) and mostly confined to exception-controlled C++ code with RAII for lock management. If C++ is OK, I can write and post a tool for evaluation, but if it must be plain C ... well, I'll avoid scarring you with the sight of what I'd produce. I suspect that a custom tool for this job could actually be fairly simple. A lot simpler than a proper, clean and usable parallel psql, anyway. -- Craig Ringer
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 >> Doing this without DBI is going to be ten times harder than doing it >> with DBI. Are we really sure that's not a viable option? > In the buildfarm? Yes, I think so. The philosophy of the buildfarm is > that it should do what you would do yourself by hand. > > And adding DBI as a requirement for running a buildfarm member would be > a significant extra barrier to entry, ISTM. (I am very fond of DBI, and > use it frequently, BTW) What about something less than a requirement then? If you have it great, you can run these extra tests. If you don't have it, no harm, no foul. We could even bundle DBI and DBD::Pg to ensure that the minimum versions are there. All the prerequisites should be in place for 99% of the machines: a C compiler and Perl are the biggies, and I can't see any buildfarm members running without those. :) - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001071014 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktF+ucACgkQvJuQZxSWSsjYOgCglyLIyGCr60og+iQSnyRgkowd +lYAnRDjPe/XxC7gb9OBPdpZlqU0wncK =kPIR -----END PGP SIGNATURE-----
On 2010-01-07 11:50 +0200, Craig Ringer wrote: > On 7/01/2010 9:15 AM, Robert Haas wrote: >> Doing this without DBI is going to be ten times harder than doing it >> with DBI. Are we really sure that's not a viable option? > > At this point, I'm personally wondering if it's worth putting together a > simple (ish) C program that reads a file describing command > interleavings on n connections. It fires up one thread per connection > required, then begins queuing commands up for the threads to execute in > per-thread fifo queues. The input file may contain synchronization > points where two or more explicitly specified threads (or just all > threads) must finish all their queued work before they may be given more. > CONN conn1: dbname=regress, user=regress > CONN conn2: dbname=regress, user=regress > STMT conn1: SELECT blah blah; > STMT conn2: UPDATE blah blah; > SYNC conn1, conn2 > > etc. Or alternately one-file-per-connection (which would be handy if one > connection has *lots* of commands and others only occasional ones) - the > only trouble there being how to conveniently specify synchronization points. I had a similar syntax in mind, but instead of using threads, just execute the file in order using asynchronous connections. Regards, Marko Tiikkaja
On 2010-01-07 18:13 +0200, Marko Tiikkaja wrote: > I had a similar syntax in mind, but instead of using threads, just > execute the file in order using asynchronous connections. I completely failed to make the point here which was to somehow mark which statements will (or, should) block. So here we go: A=> BEGIN; B=> BEGIN; A=> UPDATE foo ..; &B=> UPDATE foo ..; -- this will block A=> COMMIT; B=> SELECT * FROM foo; B=> COMMIT; [expected output here] Regards, Marko Tiikkaja
On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: > We have not yet fully accepted the notion that you must have Perl to > build (and, in fact, I am right now trying to verify that you don't). > I don't think that requiring Perl to test is going to fly. I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. Best, David
"Greg Sabino Mullane" <greg@turnstep.com> writes:
> We could even bundle DBI and DBD::Pg to ensure that the minimum versions
> are there.
As a packager, my reaction to that is "over my dead body". We have
enough trouble keeping our own software up to date, and pretty much
every external component that we've started to bundle has been a
disaster from a maintenance standpoint. (Examples: the zic database
is constant work and yet almost never up to date; the snowball stemmer
never gets updated.)
regards, tom lane
David E. Wheeler wrote: > On Jan 6, 2010, at 6:26 PM, Tom Lane wrote: > > >> We have not yet fully accepted the notion that you must have Perl to >> build (and, in fact, I am right now trying to verify that you don't). >> I don't think that requiring Perl to test is going to fly. >> > > I believe that the build farm already requires Perl, regardless of whether the PostgreSQL build itself requires it. > > > Unless I am mistaken, Perl is required in any case to build from CVS, although not from a tarball. DBI/DBD::pg is quite another issue, however. I have been deliberately very conservative about what modules to require for the buildfarm, and we have similarly (and I think wisely) been conservative about what modules to require for Perl programs in the build process. Using DBI/DBD::Pg would raise another issue - what version of libpq would it be using? Not the one in the build being tested, that's for sure. If you really want to use Perl then either a Pure Perl DBI driver (which Greg has talked about) or a thin veneer over libpq such as we used to have in contrib seems a safer way to go. cheers
On Thu, Jan 7, 2010 at 11:46 AM, Andrew Dunstan <andrew@dunslane.net> wrote: > Using DBI/DBD::Pg would raise another issue - what version of libpq would it > be using? Not the one in the build being tested, that's for sure. If you > really want to use Perl then either a Pure Perl DBI driver (which Greg has > talked about) or a thin veneer over libpq such as we used to have in contrib > seems a safer way to go. I completely agree. As between those two options, count me as +1 for the thin veneer. ...Robert
On Wed, Jan 06, 2010 at 08:40:28PM -0500, Andrew Dunstan wrote: > A parallel psql seems to me a better way to go. We talked about that > a while ago, but I don't recall what happened to it. Greg Stark had a patch a couple of years ago. Dunno what happened to it since then. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Jan 6, 2010, at 6:31 PM, Kevin Grittner wrote: > As far as I've been able to determine so far, to call psql in a > relatively portable way would require something like this: > > http://perldoc.perl.org/perlfork.html Here's an example using IPC::Open3: #!/usr/local/bin/perl -w use strict; use warnings; use IPC::Open3; use Symbol 'gensym'; use constant EOC => "__DONE__\n"; my ($in1, $out1, $err1) = (gensym, gensym, gensym); my ($in2, $out2, $err2) = (gensym, gensym, gensym); my $pid1 = open3 $in1, $out1, $err1, 'bash'; my $pid2 = open3 $in2, $out2, $err2, 'bash'; print $in1 "cd ~/dev/postgresql\n"; print $in1 "ls doc\n"; print $in1 "echo ", EOC; while ((my $line = <$out1>)){ last if $line eq EOC; print "LS: $line"; } print "#### Finished file listing\n\n"; print $in2 "cd ~/dev/postgresql\n"; print $in2 "head -4 README\n"; print $in2 "echo ", EOC; while (defined( my $line= <$out2> )) { last if $line eq EOC; print "HEAD: $line"; } print "#### Finished reading README\n\n"; print $in1 "exit\n"; print $in2 "exit\n"; waitpid $pid2, 0; print "#### All done!\n"; With that, I get: LS: KNOWN_BUGS LS: MISSING_FEATURES LS: Makefile LS: README.mb.big5 LS: README.mb.jp LS: TODO LS: bug.template LS: src #### Finished file listing HEAD: PostgreSQL Database Management System HEAD: ===================================== HEAD: HEAD: Thisdirectory contains the source code distribution of the PostgreSQL #### Finished reading README #### All done! I could easily write a very simple module to abstract all that stuff for you, then you could just do something like: my $psql1 = Shell::Pipe->new(qw(psql -U postgres)); my $psql2 = Shell::Pipe->new(qw(psql -U postgres)); $psql1->print('SELECT * from pg_class'); while (my $line = $psql1->readln) { print "Output: $line\n" } $psql1->close; All I'd need is some more reliable way than "echo "DONE__\n" to be able to tell when a particular command has finished outputting. Thoughts? Best, David
Andrew Dunstan <andrew@dunslane.net> writes:
> Unless I am mistaken, Perl is required in any case to build from CVS,
> although not from a tarball.
Right, but to my mind "building from a tarball" needs to include the
ability to run the regression tests on what you built. So injecting
Perl into that is moving the goalposts on build requirements.
regards, tom lane
On Jan 7, 2010, at 9:08 AM, Tom Lane wrote: > Right, but to my mind "building from a tarball" needs to include the > ability to run the regression tests on what you built. So injecting > Perl into that is moving the goalposts on build requirements. In that case, there's nothing for it except concurrent psql. Or else some sort of shell environment that's available on allplatforms. do we require bash on Windows? Oh, wait, the Windows build requires Perl… Best, David
"David E. Wheeler" <david@kineticode.com> writes:
> On Jan 7, 2010, at 9:08 AM, Tom Lane wrote:
>> Right, but to my mind "building from a tarball" needs to include the
>> ability to run the regression tests on what you built. So injecting
>> Perl into that is moving the goalposts on build requirements.
> In that case, there's nothing for it except concurrent psql.
Unless we are prepared to define concurrency testing as something
separate from the basic regression tests. Which is kind of annoying but
perhaps less so than the alternatives. It certainly seems to me to
be the kind of thing you wouldn't need to test in order to have
confidence in a local build.
regards, tom lane
On Jan 7, 2010, at 9:19 AM, Tom Lane wrote: >> In that case, there's nothing for it except concurrent psql. > > Unless we are prepared to define concurrency testing as something > separate from the basic regression tests. Which is kind of annoying but > perhaps less so than the alternatives. It certainly seems to me to > be the kind of thing you wouldn't need to test in order to have > confidence in a local build. I was rather assuming that was what we were talking about here, since we have in the past discussed testing things like dumpand restore, which would require something like Perl to handle multiple processes, and wouldn't work very well for aregular release. I think if we have the ability to add tests that are not distributed, it gives us a lot more freedom and opportunity to testthings that are not currently well-tested. Best, David
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 > Unless we are prepared to define concurrency testing as something > separate from the basic regression tests. Which is kind of annoying but > perhaps less so than the alternatives. It certainly seems to me to > be the kind of thing you wouldn't need to test in order to have > confidence in a local build. I thought we were leaning towards something separate. - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001071234 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktGGxYACgkQvJuQZxSWSsgG0gCfZ4eTpXd/97p3zSJpLqGhKMh6 8nMAoJ2lQUaCWNVeSPDU8fq7VnkO0s4C =xBOo -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 > Using DBI/DBD::Pg would raise another issue - what version of libpq > would it be using? Not the one in the build being tested, that's for > sure. Er...why not? That's what psql uses. As for those advocating using a custom C program written using libpq - that's basically what DBI/DBD::Pg ends up being! Only with a shiny Perl outside and years of real-world testing and usage. > If you really want to use Perl then either a Pure Perl DBI driver > (which Greg has talked about) or a thin veneer over libpq such as we > used to have in contrib seems a safer way to go. I'm still *very* interested in making a libpq-less pure perl driver, if anyone feels like funding it, let me know! :) - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001071236 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktGHF0ACgkQvJuQZxSWSsiFmACfUJbRDUJGvDTJNjgj/dyQKVCA tZwAn2fiXKNWbWzYXobrHZjeE8aSSiVv =sGzK -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 >> We could even bundle DBI and DBD::Pg to ensure that the minimum versions >> are there. > As a packager, my reaction to that is "over my dead body". We have > enough trouble keeping our own software up to date, and pretty much > every external component that we've started to bundle has been a > disaster from a maintenance standpoint. (Examples: the zic database > is constant work and yet almost never up to date; the snowball stemmer > never gets updated.) As a counterargument, I'll point out that this won't be as critical as zic, especially if we're talking about an additional/optional set of tests. Also, Tim Bunce and I are right here, so the maintenance should not be that bad (and I'd hazard that a lot more people in the community know Perl/DBI than zic or stemmers). - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001071315 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktGJNQACgkQvJuQZxSWSshoHwCg9urTTT19m55soiIjuYKKuB5L PjYAoJbDAe7j4xsxsSFfVEkYDFpTjKE9 =48oW -----END PGP SIGNATURE-----
Alvaro Herrera <alvherre@commandprompt.com> wrote: > Kevin Grittner escribió: > >> Are we anywhere close to an agreement on what the multi-session >> psql implementation would look like? > See > http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us > and followups. Thanks, I had missed or forgotten this thread. It seems like it drifted more-or-less to a consensus. Does everyone agree that this thread represents the will of the community? I see the thread started at the point where Greg Stark had a patch, but by the end it was a "start over, stealing code as appropriate" situation. Has anything happened with that? Any plans? -Kevin
On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote: > I'm still *very* interested in making a libpq-less pure perl driver, > if anyone feels like funding it, let me know! :) You mean this one: http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm ? Cheers, M
A.M. wrote: > On Jan 7, 2010, at 12:39 PM, Greg Sabino Mullane wrote: > >> I'm still *very* interested in making a libpq-less pure perl driver, >> if anyone feels like funding it, let me know! :) >> > > You mean this one: > > http://search.cpan.org/~arc/DBD-PgPP-0.07/lib/DBD/PgPP.pm > > ? > > > It has a list of limitations as long as your arm. cheers andrew
On 8/01/2010 1:39 AM, Greg Sabino Mullane wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: RIPEMD160 > > >> Using DBI/DBD::Pg would raise another issue - what version of libpq >> would it be using? Not the one in the build being tested, that's for >> sure. > > Er...why not? That's what psql uses. Because you'd have to build DBD::Pg against the new libpq, as you do psql. That means you need DBD::Pg sources and the build environment for Perl (headers etc) not just a working Perl runtime. Big difference. There's no guarantee the user's machine has the same major version of libpq and thus no guarantee that DBD::Pq can be redirected to use your custom libpq by LD_LIBRARY_PATH. It might also override the library search path with rpath linking. Building your own would be pretty much unavoidable unless you're prepared to either require the user to provide a matching version of DBD::Pg or have the tests running with whatever random version happens to be lying around. Using whatever DBD::Pg version happens to be present on the machine would be a bit of a nightmare for reproducibility of test results, and would be really unattractive for use in the standard tests. "make check fails on my some-random-distro" would become painfully common on the GENERAL list... Is bundling a Perl module in the source tree and building it as part of the Pg build a reasonable choice? Personally, I don't think so. Additionally, a dedicated testing tool like some folks have been talking about would be really handy for users who want to test their schema. I've had to write my own (in Java, or I'd be offering it) for this purpose, as psql is completely unsuitable for concurrent-run testing and I needed to show that my locking was safe and deadlock-free in some of the more complex stored procs I have. -- Craig Ringer
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 >> Using DBI/DBD::Pg would raise another issue - what version of libpq >> would it be using? Not the one in the build being tested, that's for >> sure. > > Er...why not? That's what psql uses. > Because you'd have to build DBD::Pg against the new libpq, as you do > psql. That means you need DBD::Pg sources and the build environment for > Perl (headers etc) not just a working Perl runtime. Big difference. Yes, but that is what I was envisioning. As you point out, that's the only sane way to make sure we have a good version of DBD::Pg with which to test. As a side effect, it put libpq through some extra paces as well. :) > Is bundling a Perl module in the source tree and building it as part of > the Pg build a reasonable choice? Personally, I don't think so. *shrug* It's different, but it's the best solution to the problem at hand. It wouldn't be built as part of Pg, only as part of the tests. > Additionally, a dedicated testing tool like some folks have been talking > about would be really handy for users who want to test their schema. > I've had to write my own (in Java, or I'd be offering it) for this > purpose, as psql is completely unsuitable for concurrent-run testing and > I needed to show that my locking was safe and deadlock-free in some of > the more complex stored procs I have. Sure, but it's the difference between waiting for someone to write something (and then dealing with the invevitable bugs, tweaks, and enhancements), or using a solid, known quantity (DBI + Test::More). - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001102316 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktKplIACgkQvJuQZxSWSsj7TQCeMOXWS+uLIZE9QbeBWPxYv/rg HhEAn0QZUzE2/8uyg5Oi+K8qL/oTeDSO =R8Az -----END PGP SIGNATURE-----
On Mon, Jan 11, 2010 at 04:17:42AM -0000, Greg Sabino Mullane wrote: > > Because you'd have to build DBD::Pg against the new libpq, as you do > > psql. That means you need DBD::Pg sources and the build environment for > > Perl (headers etc) not just a working Perl runtime. Big difference. > > Yes, but that is what I was envisioning. As you point out, that's the > only sane way to make sure we have a good version of DBD::Pg with > which to test. As a side effect, it put libpq through some extra > paces as well. :) Is there a reason why you're suggesting using DBI? There is also the Pg perl module which works as well and is one tenth of the size. It also doesn't have external dependancies. It's just a plain wrapper around libpq, which for the purposes of testing may be better. http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 > Is there a reason why you're suggesting using DBI? There is also the Pg > perl module which works as well and is one tenth of the size. It also > doesn't have external dependancies. It's just a plain wrapper around > libpq, which for the purposes of testing may be better. > > http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm Works as well? Did you take a look at that link? The last update was early 2000, which should give you an indication of just how dead it is. - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001111241 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktLYtwACgkQvJuQZxSWSsjNJgCg1IZBNIZsUMZ2V83/NjgJnrGO 3NEAoK/w0byO45zmni/i9lnhliD4UpkU =ndmb -----END PGP SIGNATURE-----
On Mon, Jan 11, 2010 at 05:42:08PM -0000, Greg Sabino Mullane wrote: > > Is there a reason why you're suggesting using DBI? There is also the Pg > > perl module which works as well and is one tenth of the size. It also > > doesn't have external dependancies. It's just a plain wrapper around > > libpq, which for the purposes of testing may be better. > > > > http://search.cpan.org/~mergl/pgsql_perl5-1.9.0/Pg.pm > > Works as well? Did you take a look at that link? The last update was > early 2000, which should give you an indication of just how dead > it is. Dead or not, it still works, even against 8.4. I have many programs that use it. It's simply a wrapper around the libpq interface and as long as the libpq interface remains stable (which we go to great pains to do), so will this module. Given the talk of importing some perl module into the postgresql tree it just seemed more logical to me to take something that was close to libpq and had no external dependancies than taking a module with an external dependancy (namely DBI). Have a nice day, -- Martijn van Oosterhout <kleptog@svana.org> http://svana.org/kleptog/ > Please line up in a tree and maintain the heap invariant while > boarding. Thank you for flying nlogn airlines.
Hi,
Markus Wanner wrote:
> Sorry, if that didn't get clear. I'm trying to put together something I
> can release real soon now (tm). I'll keep you informed.
Okay, here we go: dtester version 0.0.
This emerged out of Postgres-R, where I don't just need to test multiple
client connections, but multiple postmasters interacting with each
other. None the less, it may be suitable for other needs as well,
especially testing with concurrent sessions.
I've decided to release this as a separate project named dtester, as
proposed by Michael Tan (thanks for your inspiration).
It's certainly missing lots of things, mainly documentation. However,
I've attached a patch which integrates nicely into the Postgres
Makefiles, so you just need to say: make dcheck.
That very same patch includes a test case with three concurrent
transactions with circular dependencies, where the current SERIALIZABLE
isolation level fails to provide serializability.
Installing dtester itself is as simple as 'python setup.py install' in
the extracted archive's directory.
Go try it, read the code and simply ask, if you get stuck. I'll try to
come up with some more documentation and such...
Regards
Markus Wanner
#
# old_revision [651e6b451b14a926bbca5ac0b308bc5f979c8c8e]
#
# add_file "src/test/regress/pg_dtester.py.in"
# content [edcf9857ee4d884b6120b72bf319774126241ee3]
#
# patch "GNUmakefile.in"
# from [da68d137cee6cec6458a1d9e877c5b623a11415a]
# to [b605a66cee8f77978da09c326cf9dbaeba464bd2]
#
# patch "src/test/regress/GNUmakefile"
# from [ed08fe1be025ede31fa2dba35f81f653809a2096]
# to [55ae74cf991ae07b0d5b082882ff5ba1f1ef4036]
#
============================================================
--- src/test/regress/pg_dtester.py.in edcf9857ee4d884b6120b72bf319774126241ee3
+++ src/test/regress/pg_dtester.py.in edcf9857ee4d884b6120b72bf319774126241ee3
@@ -0,0 +1,762 @@
+#!/usr/bin/python
+
+#-------------------------------------------------------------------------
+#
+# dtester.py.in
+#
+# Sample test suite running two concurrent transactions, showing
+# off some capabilities of dtester.
+#
+# Copyright (c) 2006-2010, Markus Wanner
+#
+#-------------------------------------------------------------------------
+
+import re, os, sys, getopt
+from twisted.internet import defer, reactor
+
+from dtester.events import EventMatcher, EventSource, Event, \
+ ProcessOutputEvent, ProcessErrorEvent, ProcessEndedEvent
+from dtester.exceptions import TestAborted
+from dtester.test import TestSuite, BaseTest, SyncTest
+from dtester.reporter import StreamReporter
+from dtester.runner import Runner, Timeout
+
+# ****** definition of tests and suites ***********************************
+
+class InstallationSuite(TestSuite):
+
+ setUpDescription = "creating temporary installation"
+ tearDownDescription = "removing temporary installation"
+
+ needs = (('shell', "IShell or something"),)
+
+ def setUp(self):
+ # inherit getConfig from the shell
+ setattr(self, 'getConfig', self.shell.getConfig)
+ setattr(self, 'runCommand', self.shell.runCommand)
+ setattr(self, 'recursive_remove', self.shell.recursive_remove)
+
+ # (re) create an installation directory
+ self.pg_inst_dir = self.shell.getConfig('inst_dir')
+ if os.path.exists(self.pg_inst_dir):
+ self.shell.recursive_remove(self.pg_inst_dir)
+ os.mkdir(self.pg_inst_dir)
+
+ # install into that directory
+ proc = self.shell.runCommand('make', 'make',
+ args=['make', '-C', self.shell.getConfig('top-builddir'),
+ 'DESTDIR=%s' % self.pg_inst_dir, 'install',
+ 'with_perl=no', 'with_python=no'],
+ lineBasedOutput=True)
+
+ d = self.waitFor(proc, EventMatcher(ProcessEndedEvent))
+ d.addCallback(self.makeTerminated)
+ proc.start()
+
+ # FIXME: how to properly handle these?
+ self.shell.addEnvPath(self.shell.getConfig('bindir'))
+ self.shell.addEnvLibraryPath(self.shell.getConfig('libdir'))
+ return d
+
+ def makeTerminated(self, event):
+ if event.exitCode != 0:
+ raise Exception("Initdb returned %d" % event.exitCode)
+ else:
+ return True
+
+ def tearDown(self):
+ # The installation procedure should be able to simply override any
+ # formerly installed files, so we save the time to clean up the
+ # installation directory.
+ return
+
+
+class InitdbSuite(TestSuite):
+
+ args = (('number', int), )
+ needs = (('shell', "IShell or something"),)
+
+ def setUpDescription(self):
+ return "initializing database system %d" % self.number
+
+ def tearDownDescription(self):
+ return "removing database system %d" % self.number
+
+ def getNumber(self):
+ return self.number
+
+ def getDir(self):
+ return self.dbdir
+
+ def setUp(self):
+ self.dbdir = "%s%d" % \
+ (self.shell.getConfig('pgdata_prefix'), self.number)
+ proc = self.shell.runCommand(
+ 'initdb-%d' % self.number,
+ 'initdb', args = [
+ 'initdb', '-D', self.dbdir,
+ '-A', 'trust', '--noclean'],
+ lineBasedOutput=True)
+
+ d = defer.Deferred()
+ proc.addHook(EventMatcher(ProcessEndedEvent),
+ self.initdb_terminated, d)
+ proc.start()
+ return d
+
+ def initdb_terminated(self, event, d):
+ if event.exitCode != 0:
+ d.errback(Exception("Initdb returned %d" % event.exitCode))
+ else:
+ d.callback(True)
+
+ def tearDown(self):
+ self.shell.recursive_remove(
+ "%s%d" % (self.shell.getConfig('pgdata_prefix'), self.number))
+
+
+class PostmasterSuite(TestSuite):
+
+ needs = (('shell', "IShell or something"),
+ ('dbdir', "IDatabaseDir"),)
+
+ def setUpDescription(self):
+ return "starting database system %d" % self.dbdir.getNumber()
+
+ def tearDownDescription(self):
+ return "stopping database system %d" % self.dbdir.getNumber()
+
+ def getPort(self):
+ return self.port
+
+ def setUp(self):
+ setattr(self, 'getNumber', self.dbdir.getNumber)
+
+ self.port = self.shell.getConfig('temp-port') + self.dbdir.getNumber()
+ self.postmaster = self.shell.runCommand(
+ 'postmaster%d' % self.dbdir.getNumber(),
+ 'postmaster',
+ # FIXME: -A1 doesn't exist if assertions are disabled
+ args = ['postmaster', '-A1', '-d5',
+ '-D', self.dbdir.getDir(),
+ '-i', '-p', str(self.port)],
+ lineBasedOutput=True)
+
+ d = defer.Deferred()
+ self.readyHook = \
+ self.postmaster.addHook(EventMatcher(ProcessErrorEvent,
+ "database system is ready to accept connections"),
+ self.postmaster_ready, d)
+
+ self.unexpectedTerminationHook = \
+ self.postmaster.addHook(EventMatcher(ProcessEndedEvent),
+ self.postmaster_terminated)
+ self.postmaster.start()
+ return d
+
+ def postmaster_ready(self, event, d):
+ # it's sufficient if we're called once
+ self.postmaster.removeHook(self.readyHook)
+ d.callback(None)
+
+ def postmaster_terminated(self, event):
+ exitCode = 'undef'
+ if hasattr(event, 'exitCode'):
+ exitCode = event.exitCode
+ elif hasattr(event, 'data'):
+ exitCode = repr(event.data)
+ self.abort("postmaster %d unexpectedly terminated (exit code %s)" % \
+ (self.dbdir.getNumber(), exitCode))
+
+ def tearDown(self):
+ self.postmaster.removeHook(self.unexpectedTerminationHook)
+ if not self.aborted:
+ d = defer.Deferred()
+ self.postmaster.addHook(EventMatcher(ProcessEndedEvent),
+ lambda event: d.callback(None))
+ self.postmaster.stop()
+ return d
+ else:
+ return True
+
+
+class TestDatabaseSuite(TestSuite):
+
+ args = (('dbname', str),)
+ needs = (('shell', "IShell or something"),
+ ('pg', "IPostmaster"),)
+
+ def setUpDescription(self):
+ return "creating database %s at server %d" % \
+ (self.dbname, self.pg.getNumber())
+ def tearDownDescription(self):
+ return "dropping database %s at server %d" % \
+ (self.dbname, self.pg.getNumber())
+
+ def getDbname(self):
+ return self.dbname
+
+ def setUp(self):
+ setattr(self, "getPort", self.pg.getPort)
+ setattr(self, "getNumber", self.pg.getNumber)
+
+ self.proc = self.shell.runCommand(
+ 'createdb%d' % self.pg.getNumber(),
+ 'createdb',
+ args = ['createdb',
+ '-p', str(self.getPort()), self.dbname],
+ lineBasedOutput=True)
+
+ d = defer.Deferred()
+ self.proc.addHook(EventMatcher(ProcessEndedEvent),
+ self.createdb_terminated, d)
+ self.proc.start()
+ return d
+
+ def createdb_terminated(self, event, d):
+ if event.exitCode != 0:
+ d.errback(Exception("createdb terminated with code %d" % \
+ event.exitCode))
+ else:
+ d.callback(None)
+
+ def tearDown(self):
+ if self.pg.aborted:
+ return True
+
+ # Hm.. this interferes with the postmaster suites, which need
+ # to be started and stopped several times on top of a test database,
+ # however, creating and dropping it certainly depends on a running
+ # postmaster. Not sure how to solve this, at the moment I'm just
+ # skipping cleanup, i.e. dropdb.
+ return True
+
+ self.proc = self.shell.runCommand(
+ 'dropdb%d' % self.pg.getNumber(),
+ 'dropdb',
+ args = ['dropdb',
+ '-p', str(self.getPort()), self.dbname],
+ lineBasedOutput=True)
+
+ d = defer.Deferred()
+ self.proc.addHook(EventMatcher(ProcessEndedEvent),
+ self.dropdb_terminated, d)
+ self.proc.start()
+ return d
+
+ def dropdb_terminated(self, event, d):
+ if event.exitCode != 0:
+ d.errback(Exception("dropdb returned with %d" % \
+ event.exitCode))
+ else:
+ d.callback(None)
+
+
+class SqlConnectionSuite(TestSuite):
+
+ args = (('dbname', str),)
+ needs = (('shell', "IShell or something"),
+ ('db', "IPostmaster"))
+
+ def setUpDescription(self):
+ return "connecting to database %s at server %d" % \
+ (self.dbname, self.db.getNumber())
+ def tearDownDescription(self):
+ return "disconnecting from database %s at server %d" % \
+ (self.dbname, self.db.getNumber())
+
+ def getDbname(self):
+ return self.dbname
+
+ def setUp(self):
+ self.psql = self.shell.runCommand(
+ 'psql%d' % self.db.getNumber(),
+ 'psql',
+ args=['psql', '-A',
+ '-p', str(self.db.getPort()), self.dbname])
+
+ # initialize the output buffer and attach a first output collector
+ # *before* the process is started.
+ self.output_buffer = ""
+ d = defer.Deferred()
+ self.outputCollectorDeferred = d
+ self.outputCollectorHook = self.psql.addHook(
+ EventMatcher(ProcessOutputEvent), self.outputCollector,
+ None, d)
+
+ # Mark as being in used, until we get to the commandline
+ self.inUse = True
+ self.workQueue = []
+
+ # also add a termination hook
+ self.unexpectedTerminationHook = self.psql.addHook(
+ EventMatcher(ProcessEndedEvent), self.psql_terminated)
+
+ # then schedule start of the psql process and return the deferred
+ # *before* starting the process.
+ reactor.callLater(0.0, self.psql.start)
+ return d
+
+ def psql_terminated(self, event):
+ exitCode = "undef"
+ if hasattr(event, 'exitCode'):
+ exitCode = event.exitCode
+ elif hasattr(event, 'data'):
+ exitCode = repr(event.data)
+
+ # If there's an outputCollectorHook, the abort method won't catch
+ # and we have to wait for the timeout to trigger, instead of
+ # acting on process termination. We thus save the outputCollector
+ # deferred and send it an errback with the failure.
+ if self.outputCollectorHook:
+ self.outputCollectorDeferred.errback( \
+ TestAborted("psql to server %d unexpectedly terminated (exit code %s)" % ( \
+ self.db.getNumber(), exitCode)))
+ self.abort(
+ "psql to server %d unexpectedly terminated (exit code %s)" % ( \
+ self.db.getNumber(), exitCode))
+
+ def tearDown(self):
+ self.psql.removeHook(self.unexpectedTerminationHook)
+
+ d = defer.Deferred()
+ self.psql.addHook(EventMatcher(ProcessEndedEvent),
+ lambda event: d.callback(None))
+ reactor.callLater(0.0, self.psql.write, "\\q\n")
+ reactor.callLater(5.0, self.psql.stop)
+ return d
+
+ def outputCollector(self, event, query, d):
+ self.output_buffer += event.data
+
+ cmdprompt = self.dbname + '=#'
+ cpos = self.output_buffer.find(cmdprompt)
+
+ if cpos >= 0:
+ self.psql.removeHook(self.outputCollectorHook)
+ self.outputCollectorHook = False
+ result = self.output_buffer[:cpos]
+ self.output_buffer = self.output_buffer[cpos + len(cmdprompt):]
+ if len(self.output_buffer) > 0 and self.output_buffer != ' ':
+ print "rest: %s" % repr(self.output_buffer)
+ if d:
+ # remove the command prompt at the end
+ result = result[:cpos]
+
+ if query:
+ # remove the query string at the beginning
+ query_len = len(query)
+ if result[:query_len] == (query):
+ result = result[query_len:]
+ while (len(result) > 1) and (result[0] in ("\n", "\r")):
+ result = result[1:]
+ reactor.callLater(0.0, d.callback, result)
+
+ self.inUse = False
+ if len(self.workQueue) > 0:
+ assert not self.inUse
+ job = self.workQueue.pop()
+ d1 = job['method'](*job['args'])
+ d1.chainDeferred(job['deferred'])
+
+ def query(self, query):
+ if self.inUse:
+ d = defer.Deferred()
+ self.workQueue.append({'deferred': d,
+ 'method': self.query,
+ 'args': (query,)})
+ return d
+
+ assert not self.inUse
+ assert not self.outputCollectorHook
+
+ self.inUse = True
+ self.output_buffer = ""
+ d = defer.Deferred()
+ self.outputCollectorHook = self.psql.addHook(
+ EventMatcher(ProcessOutputEvent), self.outputCollector, query, d)
+ d.addCallback(self.parseQueryResult)
+
+ # defer writing to the process, so that the caller has the
+ # opportunity to add callbacks to the deferred we return.
+ reactor.callLater(0.0, self.psql.write, query + "\n")
+
+ return d
+
+ def parseQueryResult(self, result):
+ lines = result.split('\n')
+ # strip empty newlines at the end
+ while len(lines[-1].strip()) == 0:
+ lines = lines[:-1]
+ #print "lines: %s" % lines
+
+ try:
+ assert len(lines) >= 2
+
+ lines = map(lambda x: x.strip(), lines)
+ headLine = lines[0]
+ tailLine = lines[-1]
+
+ fields = headLine.split('|')
+ rows = []
+ for row in lines[1:-1]:
+ attrs = row.split('|')
+ assert len(attrs) == len(fields)
+ x = {}
+ for i in range(len(attrs)):
+ x[fields[i]] = attrs[i].strip()
+ rows.append(x)
+
+ x = re.compile("\((\d+) rows?\)").search(tailLine)
+ if x:
+ if not int(x.group(1)) == len(rows):
+ print "number of rows doesn't match: %s vs %d" % (
+ x.group(1), len(rows))
+ print "for: %s" % lines
+ else:
+ raise Exception("final number of rows line doesn't match.\n------------\n%s\n---------------\n" %
lines)
+ return rows
+ except Exception, e:
+ import traceback
+ print "error parsing query result: %s" % e
+ traceback.print_exc()
+ raise e
+ # return []
+
+ def operation(self, query, expResult):
+ if self.inUse:
+ d = defer.Deferred()
+ self.workQueue.append({'deferred': d,
+ 'method': self.operation,
+ 'args': (query, expResult)})
+ return d
+
+ assert not self.inUse
+ assert not self.outputCollectorHook
+
+ self.inUse = True
+ self.output_buffer = ""
+ d = defer.Deferred()
+ self.outputCollectorDeferred = d
+ self.outputCollectorHook = self.psql.addHook(
+ EventMatcher(ProcessOutputEvent), self.outputCollector, query, d)
+ d.addCallback(self.checkQueryResult, expResult)
+
+ # defer writing to the process, so that the caller has the
+ # opportunity to add callbacks to the deferred we return.
+ reactor.callLater(0.0, self.psql.write, query + "\n")
+
+ return d
+
+ def checkQueryResult(self, result, expResult):
+ x = re.compile("^" + expResult, re.M).search(result)
+ if not x:
+ print "result:\n---------\n%s\n----------\n" % result
+ print "expResult:\n---------\n%s\n----------\n" % expResult
+ print "WRONG RESULT!"
+ raise Exception("didn't get expected result")
+ return result
+
+
+class TestDatabaseConnection(BaseTest):
+
+ needs = (('conn', "ISqlConnection"),)
+
+ description = "database connection"
+
+ def run(self):
+ return self.conn.query("SELECT 1 AS test;")
+
+
+# FIXME: that's not actually a test, but it modifies the database state
+class PopulateTestDatabase(BaseTest):
+
+ needs = (('conn', "ISqlConnection"),)
+
+ description = "populate test database"
+
+ def run(self):
+ conn = self.conn
+
+ # Create a test table for use in TestConcurrentUpdates and fill it
+ # with two test tuples.
+ d = conn.operation("CREATE TABLE test (i int PRIMARY KEY, t text);",
+ "CREATE TABLE")
+ d.addCallback(lambda x: conn.operation(
+ "INSERT INTO test VALUES (5, 'apple');",
+ "INSERT"))
+ d.addCallback(lambda x: conn.operation(
+ "INSERT INTO test VALUES (7, 'pear');",
+ "INSERT"))
+ d.addCallback(lambda x: conn.operation(
+ "INSERT INTO test VALUES (11, 'banana');",
+ "INSERT"))
+ return d
+
+
+class TestTrueSerializabilityConcurrentUpdates(SyncTest):
+ """ Runs three transactions concurrently, each reading from what the
+ other writes in turn. Should raise a serialization failure, but
+ instead leads to wrong results, ATM.
+ """
+
+ description = "concurrent updates"
+
+ needs = (('conn1', 'ISqlConnection'),
+ ('conn2', 'ISqlConnection'),
+ ('conn3', 'ISqlConnection'))
+
+ def execOnAllConnections(self, sql, expRes):
+ deferreds = []
+ for conn in self.connections:
+ d = conn.operation(sql, expRes)
+ deferreds.append(d)
+
+ d = defer.DeferredList(deferreds,
+ consumeErrors=True, fireOnOneErrback=True)
+ return d
+
+ def readValueThenWrite(self, conn, readFromId, writeToId):
+ d = conn.query("SELECT t FROM test WHERE i = %d;" % readFromId)
+ d.addCallback(self.writeValueBack, conn, writeToId)
+ return d
+
+ def writeValueBack(self, result, conn, writeToId):
+ assert len(result) == 1
+ row = result[0]
+ assert len(row) == 1
+ value = row['t']
+ d = conn.operation("UPDATE test SET t = '%s' WHERE i = %d;" % (value, writeToId),
+ "UPDATE")
+ return d
+
+ def startConcurrentOperations(self):
+ d1 = self.readValueThenWrite(self.conn1, readFromId=5, writeToId=7)
+ d2 = self.readValueThenWrite(self.conn2, readFromId=7, writeToId=11)
+ d3 = self.readValueThenWrite(self.conn3, readFromId=11, writeToId=5)
+ return defer.DeferredList([d1, d2, d3],
+ consumeErrors=False, fireOnOneErrback=True)
+
+ def run(self):
+ self.connections = [
+ self.conn1,
+ self.conn2,
+ self.conn3]
+
+ # begin a transaction on all three connections
+ self.syncCall(10, self.execOnAllConnections,
+ "BEGIN;", "BEGIN")
+
+ # set their isolation level to SERIALIZABLE
+ self.syncCall(10, self.execOnAllConnections,
+ "SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;", "SET")
+
+ # concurrently let each of the three transactions read a value and
+ # write that to another tuple, wait for all the UPDATEs to complete
+ # before trying to commit any of the transactions
+ self.syncCall(10, self.startConcurrentOperations)
+
+ # try to commit all three transactions (accepting both COMMIT or
+ # ERROR, we check the result later on).
+ self.syncCall(10, self.execOnAllConnections,
+ "COMMIT;", "COMMIT|ERROR");
+
+ # count the occurrance of each fruit
+ result = self.syncCall(10, self.conn1.query,
+ "SELECT t FROM test WHERE i IN (5, 7, 11);")
+ counters = {'banana': 0, 'apple': 0, 'pear': 0}
+ for row in result:
+ counters[row['t']] += 1
+
+ # you currently get one fruit each, as no transaction gets aborted,
+ # which is impossible if the transactions had been executed one
+ # after another.
+ self.assertNotEqual(counters.values(), [1, 1, 1])
+
+class TestTrueSerializabilityConcurrentInsert(SyncTest):
+ """ Runs three transactions concurrently, each reading from what the
+ other writes in turn. Should raise a serialization failure, but
+ instead leads to wrong results, ATM.
+ """
+
+ description = "concurrent insert"
+
+ needs = (('conn1', 'ISqlConnection'),
+ ('conn2', 'ISqlConnection'))
+
+ def execOnAllConnections(self, sql, expRes):
+ deferreds = []
+ for conn in self.connections:
+ d = conn.operation(sql, expRes)
+ deferreds.append(d)
+
+ d = defer.DeferredList(deferreds,
+ consumeErrors=True, fireOnOneErrback=True)
+ return d
+
+ def run(self):
+ self.connections = [
+ self.conn1,
+ self.conn2]
+
+ # begin a transaction on all three connections
+ self.syncCall(10, self.execOnAllConnections,
+ "BEGIN;", "BEGIN")
+
+ # set their isolation level to SERIALIZABLE
+ self.syncCall(10, self.execOnAllConnections,
+ "SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;", "SET")
+
+ # try to commit all three transactions (accepting both COMMIT or
+ # ERROR, we check the result later on).
+ self.syncCall(10, self.execOnAllConnections,
+ "COMMIT;", "COMMIT|ERROR");
+
+
+# ****** test running code ************************************************
+
+class Logger(object):
+ """ A simplistic logger that just writes it all into one single file.
+ """
+ def __init__(self, logFileName):
+ self.logfile = open(logFileName, 'w')
+
+ def __del__(self):
+ self.logfile.close()
+
+ def callback(self, event):
+ self.logfile.write(str(event) + "\n")
+ self.logfile.flush()
+
+def main(argv):
+ print "Postgres dtester suite Copyright (c) 2004-2010, by Markus Wanner\n"
+
+ config = {
+ 'temp-port': 65432,
+
+ # by default, use the same installation directory as make check
+ 'inst_dir': os.path.join(os.getcwd(), 'tmp_check/install'),
+
+ # and a similar prefix
+ 'pgdata_prefix': os.path.join(os.getcwd(), 'tmp_check/data-dtester'),
+ 'logfile' : os.path.join(os.getcwd(), 'dtester.log'),
+ }
+
+ try:
+ opts, args = getopt.getopt(argv,
+ "h",
+ ["help", "temp-install", "top-builddir=", "temp-port=",
+ "multibyte="])
+ except getopt.GetoptError:
+ usage()
+ sys.exit(2)
+
+ for opt, arg in opts:
+ if opt in ("-h", "--help"):
+ usage()
+ sys.exit()
+ elif opt in ("--temp-install"):
+ config["temp-install"] = True
+ elif opt in ("--temp-port"):
+ try:
+ arg = int(arg)
+ if arg >= 1024 and arg <= 65535:
+ config["temp-port"] = arg
+ else:
+ print "temp-port out of range."
+ sys.exit(2)
+ except ValueError:
+ print "Fatal: invalid temp-port specified"
+ sys.exit(2)
+ elif opt in ("--top-builddir"):
+ config["top-builddir"] = arg
+
+
+ if not config.has_key('bindir'):
+ bindir = '@bindir@'
+ if bindir[0] == '/':
+ bindir = bindir[1:]
+ config['bindir'] = os.path.join(config['inst_dir'], bindir)
+ if not config.has_key('libdir'):
+ libdir = '@libdir@'
+ if libdir[0] == '/':
+ libdir = libdir[1:]
+ config['libdir'] = os.path.join(config['inst_dir'], libdir)
+ if not config.has_key('datadir'):
+ datadir = '@datadir@'
+ if datadir[0] == '/':
+ datadir = datadir[1:]
+ config['datadir'] = os.path.join(config['inst_dir'], datadir)
+
+
+ # FIXME: should not have to be here
+ logger = Logger(config['logfile'])
+ config['main_logging_hook'] = (EventMatcher(Event), logger.callback)
+
+
+ # definition of tests and suites, including their dependencies
+ tdef = {
+ # runs 'make install' to make sure the installation is up to date
+ 'temp_install': {'class': InstallationSuite,
+ 'uses': ('__system__',)},
+
+ # runs initdb, providing the Postgres data directory
+ 'initdb-0': {'class': InitdbSuite,
+ 'uses': ('temp_install',),
+ 'args': (0,)},
+
+ # runs a postmaster on the created database directory
+ 'pg-0': {'class': PostmasterSuite,
+ 'uses': ('temp_install', 'initdb-0')},
+
+ # creates a test database on pg-0
+ 'testdb': {'class': TestDatabaseSuite,
+ 'uses': ('temp_install', 'pg-0'),
+ 'args': ('testdb',)},
+
+ # open two connections
+ 'conn-0A': {'class': SqlConnectionSuite,
+ 'uses': ('temp_install', 'pg-0'),
+ 'args': ('testdb',),
+ 'depends': ('testdb',)},
+ 'conn-0B': {'class': SqlConnectionSuite,
+ 'uses': ('temp_install', 'pg-0'),
+ 'args': ('testdb',),
+ 'depends': ('testdb',)},
+ 'conn-0C': {'class': SqlConnectionSuite,
+ 'uses': ('temp_install', 'pg-0'),
+ 'args': ('testdb',),
+ 'depends': ('testdb',)},
+
+ # test the connections
+ 'test-conn-0A': {'class': TestDatabaseConnection,
+ 'uses': ('conn-0A',)},
+ 'test-conn-0B': {'class': TestDatabaseConnection,
+ 'uses': ('conn-0B',)},
+ 'test-conn-0C': {'class': TestDatabaseConnection,
+ 'uses': ('conn-0C',)},
+
+ # populate the test database
+ 'populate-testdb': {'class': PopulateTestDatabase,
+ 'uses': ('conn-0A',),
+ 'onlyAfter': ('test-conn-0A', 'test-conn-0B',
+ 'test-conn-0C')},
+
+ 'ser-updates': {'class': TestTrueSerializabilityConcurrentUpdates,
+ 'uses': ('conn-0A', 'conn-0B', 'conn-0C'),
+ 'onlyAfter': ('populate-testdb',)},
+
+ 'ser-insert': {'class': TestTrueSerializabilityConcurrentInsert,
+ 'uses': ('conn-0A', 'conn-0B'),
+ 'onlyAfter': ('populate-testdb',)},
+ }
+
+
+ reporter = StreamReporter()
+ runner = Runner(reporter, testTimeout=60, suiteTimeout=180)
+ runner.run(tdef, config)
+
+
+if __name__ == "__main__":
+ main(sys.argv[1:])
+
============================================================
--- GNUmakefile.in da68d137cee6cec6458a1d9e877c5b623a11415a
+++ GNUmakefile.in b605a66cee8f77978da09c326cf9dbaeba464bd2
@@ -57,7 +57,7 @@ check: all
check: all
-check installcheck installcheck-parallel:
+check dcheck installcheck installcheck-parallel:
$(MAKE) -C src/test $@
GNUmakefile: GNUmakefile.in $(top_builddir)/config.status
============================================================
--- src/test/regress/GNUmakefile ed08fe1be025ede31fa2dba35f81f653809a2096
+++ src/test/regress/GNUmakefile 55ae74cf991ae07b0d5b082882ff5ba1f1ef4036
@@ -135,6 +135,23 @@ tablespace-setup:
##
+## Prepare for dtester tests
+##
+pg_dtester.py: pg_dtester.py.in GNUmakefile $(top_builddir)/src/Makefile.global
+ sed -e 's,@bindir@,$(bindir),g' \
+ -e 's,@libdir@,$(libdir),g' \
+ -e 's,@pkglibdir@,$(pkglibdir),g' \
+ -e 's,@datadir@,$(datadir),g' \
+ -e 's/@VERSION@/$(VERSION)/g' \
+ -e 's/@host_tuple@/$(host_tuple)/g' \
+ -e 's,@GMAKE@,$(MAKE),g' \
+ -e 's/@enable_shared@/$(enable_shared)/g' \
+ -e 's/@GCC@/$(GCC)/g' \
+ $< >$@
+ chmod a+x $@
+
+
+##
## Run tests
##
@@ -152,6 +169,11 @@ standbycheck: all
standbycheck: all
$(pg_regress_call) --psqldir=$(PSQLDIR) --schedule=$(srcdir)/standby_schedule --use-existing
+dcheck: pg_dtester.py
+ ./pg_dtester.py --temp-install --top-builddir=$(top_builddir) \
+ --multibyte=$(MULTIBYTE) $(MAXCONNOPT) $(NOLOCALE)
+
+
# old interfaces follow...
runcheck: check
Attachment
Markus Wanner <markus@bluegap.ch> wrote: > Okay, here we go: dtester version 0.0. C'mon, you could have tried to inspire a *bit* more confidence by calling it version 0.1 or something! ;-) > It's certainly missing lots of things, mainly documentation. > However, I've attached a patch which integrates nicely into the > Postgres Makefiles, so you just need to say: make dcheck. That sounds very cool. > That very same patch includes a test case with three concurrent > transactions with circular dependencies, where the current > SERIALIZABLE isolation level fails to provide serializability. Fantastic! I'll expand that a bit.... > Installing dtester itself is as simple as 'python setup.py > install' in the extracted archive's directory. > > Go try it, read the code and simply ask, if you get stuck. I'll > try to come up with some more documentation and such... I'm reading through it all now. Expect feedback soon! And THANK YOU VERY MUCH! -Kevin
Hi, Kevin Grittner wrote: > C'mon, you could have tried to inspire a *bit* more confidence by > calling it version 0.1 or something! ;-) LOL As David used to say: JFDI > I found that I just needed to ask for python-twisted. Oh, sorry, yes, requirements: python, twisted. I must admit that I haven't ever tested on python 2.6 before. I'll try that (especially as it's the staircase to 3.0, IIUC). Two more things: the concurrent update test (in the patch part) is complete, while the second one is just a skeleton, ATM. (Just does a concurrent COMMIT without actually doing anything). Second: at the very end of pg_dtester.py, you find the line: reporter = StreamReporter() Try a CursesReporter() instead, it gives much nicer output! Regards Markus Wanner
Markus Wanner <markus@bluegap.ch> wrote: > I must admit that I haven't ever tested on python 2.6 before. I'll > try that (especially as it's the staircase to 3.0, IIUC). I don't use python much, so I can't comment on that. I do see that my system has these two versions on it, with a symlink that makes 2.6 the default. Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 Python 3.0.1+ (r301:69556, Apr 15 2009, 15:59:22) [GCC 4.3.3] on linux2 I haven't quite gotten it to work yet; I'll start over with 3.0 and see how it goes. I'll also attach the results of the 2.6 attempt. > Try a CursesReporter() instead, it gives much nicer output! Thanks, I'll try it. A few other issues in testing so far: (1) I see that a 'make dcheck' does a 'make install'. That's not right. For one thing I usually install in a location where I need to sudo to install; but more importantly, I want to do all checks *before* I install. It's easy enough to work around that for now, but I don't think it's acceptable long-term. (2) After a 'make dcheck' failure, the cluster created for the testing is left running. (3) If the install could check dependencies, report problems, and refuse to install without required packages, that would be less confusing for python novices (like me). Perhaps some of these problems will go away with python 3.0, but I figured I should pass the info along. Thanks again for this. It should help me a lot. -Kevin
I wrote: > I'll also attach the results of the 2.6 attempt. Let's try that again. -Kevin
Attachment
Hi,
Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>:
> I haven't quite gotten it to work yet; I'll start over with 3.0 and
> see how it goes.
Let's stick to 2.x versions, first...
> I'll also attach the results of the 2.6 attempt.
Thanks, that looks already pretty promising. ;-)
> A few other issues in testing so far:
>
> (1) I see that a 'make dcheck' does a 'make install'. That's not
> right. For one thing I usually install in a location where I need
> to sudo to install; but more importantly, I want to do all checks
> *before* I install. It's easy enough to work around that for now,
> but I don't think it's acceptable long-term.
It does: "temp_install: creating temporary installation" means it's
running make install in the background.
> (2) After a 'make dcheck' failure, the cluster created for the
> testing is left running.
That counts as a bug. I also get that from time to time (and with
Postgres-R testing on 3+ instances, it's even more annoying).
Note that the error just before that is, that a psql process it starts
cannot connect to its postmaster ("startup of test test-conn-0A
failed, skipping.") Please check the log
(src/test/regress/dtester.log) for why that failed in the first place.
Can you connect manually to the database (that's still running after a
make dcheck)?
> (3) If the install could check dependencies, report problems, and
> refuse to install without required packages, that would be less
> confusing for python novices (like me).
I'm not exactly a distutils hacker... Anybody else got any clue here?
> Perhaps some of these problems will go away with python 3.0, but I
> figured I should pass the info along.
I'd rather suspect that more of them will arise.
Regards
Markus
"Markus Wanner" <markus@bluegap.ch> wrote:
> Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>:
>> I haven't quite gotten it to work yet; I'll start over with 3.0
>> and see how it goes.
>
> Let's stick to 2.x versions, first...
OK
> It does: "temp_install: creating temporary installation" means
> it's running make install in the background.
OK, sorry for misreading that.
>> (2) After a 'make dcheck' failure, the cluster created for the
>> testing is left running.
>
> That counts as a bug. I also get that from time to time (and with
> Postgres-R testing on 3+ instances, it's even more annoying).
>
> Note that the error just before that is, that a psql process it
> starts cannot connect to its postmaster ("startup of test
> test-conn-0A failed, skipping.") Please check the log
> (src/test/regress/dtester.log) for why that failed in the first
> place.
Not sure what's relevant there. Entire file tarball attached.
> Can you connect manually to the database (that's still running
> after a make dcheck)?
Yes I can. Any queries you'd like me to run in there?
-Kevin
Attachment
Markus Wanner <markus@bluegap.ch> wrote: > Second: at the very end of pg_dtester.py, you find the line: > reporter = StreamReporter() > > Try a CursesReporter() instead, it gives much nicer output! When I try to do that, Kate complains (I'm even copying their typo): You are trying to save a python file as non ASCII, without specifiying a correct source encoding line for encoding "utf-8" It offers these options: Insert: # -*- coding: utf-8 -*- Save Nevertheless Cancel Should that coding line be in there? -Kevin
-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 > Dead or not, it still works, even against 8.4. I have many programs > that use it. It's simply a wrapper around the libpq interface and as > long as the libpq interface remains stable (which we go to great pains > to do), so will this module. Well, I stand corrected. Good to know. > Given the talk of importing some perl module into the postgresql tree > it just seemed more logical to me to take something that was close to > libpq and had no external dependancies than taking a module with an > external dependancy (namely DBI). Yes, I could see that. Actually, I just came across another one by Hiroyuki OYAMA and Aaron Crane. This was last updated January 10, 2010! : http://search.cpan.org/~arc/DBD-PgPP-0.08/ Still requires DBI of course, but no Perl library or compiling required as DBD::Pg does. So we've not got three valid options. :) - -- Greg Sabino Mullane greg@turnstep.com PGP Key: 0x14964AC8 201001151129 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -----BEGIN PGP SIGNATURE----- iEYEAREDAAYFAktQmI0ACgkQvJuQZxSWSsgNugCgjwkT9QwGpvhcIXCNYhRcTwSW JZcAnjvrsjwpO/QvJ1LzU+cUZ4UqajxV =bu4q -----END PGP SIGNATURE-----
Hi, Kevin Grittner wrote: > Not sure what's relevant there. Entire file tarball attached. Due to reasons mentioned in this thread as well, I've decided to use psql to connect to the database. dtester is parsing its output and checks that against expectations. Hawever, that has its own pitfalls, so in the end I'm almost about to change back to using libpq or implementing the bare minimum protocol (that might have its own merits within the twisted world, if implemented in the required async fashion). Strangely, your log has escape codes in it, which I'm assuming makes the parsing choke. Is that something special to yourinstallation? My psql never colors its outputs... However, the quickest way forward probably is to filter out escape sequences. Turning off tty is not really an option, because dtester doesn't have a chance to capture all necessary events, in that mode. > Yes I can. Any queries you'd like me to run in there? It looks like psql can connect, too. It's just the parsing of outputs which fails. Regards Markus
Hi, Kevin Grittner wrote: > You are trying to save a python file as non ASCII, without > specifiying a correct source encoding line for encoding "utf-8" I wasn't aware I had non-ascii characters in there. Inserting an encoding line seems fine. I'll fix that for the upcoming version 0.1. Regards Markus
Markus Wanner <markus@bluegap.ch> wrote: > I wasn't aware I had non-ascii characters in there. Inserting an > encoding line seems fine. I'll fix that for the upcoming version > 0.1. Yeah, I couldn't find any, either. I just tried creating a minimal python file in Kate, and it gave me that even though I *know* it was all ASCII characters right off my keyboard. I guess Kate is being overly picky. On the other hand, if it silences an annoying message sometimes, maybe that's reason enough to add it. -Kevin
Markus Wanner <markus@bluegap.ch> wrote: > Strangely, your log has escape codes in it, which I'm assuming > makes the parsing choke. Is that something special to your > installation? My psql never colors its outputs... I haven't configured anything like that intentionally. I don't *see* any colors when I use psql. Can you think of anywhere I should check something which might be causing this? -Kevin
Hi, Kevin Grittner wrote: > I haven't configured anything like that intentionally. I don't > *see* any colors when I use psql. Can you think of anywhere I > should check something which might be causing this? No idea ATM. However, just to make sure that has absolutely nothing to do with the curses reporter I've written: is that dtester.log you just sent the log from a run with the StreamReporter or the CursesReporter? (Should not have any influence for the log, but you never know). Please recheck with the StreamReporter and try to grep the lines starting with "[psql0]", "[psql1]" and "[psql2]". Dtester simply logs all and any output of all 3rd party processes started. Alternatively, you may want to filter out all lines that start with "[postmaster0]", that might already reduce what we can consider noise in this case. Regards Markus Wanner
Markus Wanner <markus@bluegap.ch> wrote: > Strangely, your log has escape codes in it, which I'm assuming > makes the parsing choke. Is that something special to your > installation? My pager is "less"; could that cause it? Could the twisted environment look like one where the pager should kick in? -Kevin
Hi, Kevin Grittner wrote: > My pager is "less"; could that cause it? Could the twisted > environment look like one where the pager should kick in? Yes, that could be it. At least it fails here, too, if I set PAGER=less. Try: PAGER=more make dcheck So, the solution probably lies in adjusting the environment, before starting psql. (Maybe even dropping all existing environment variables for better control of the situation). Will add that for dtester 0.1. (Also note that I plan to move most of what's currently in the patch to the dtester package itself. However, that requires it to be (even more) generic.) Thank you for testing the tester ;-) Regards Markus
Markus Wanner wrote: > Hi, > > Kevin Grittner wrote: >> My pager is "less"; could that cause it? Could the twisted >> environment look like one where the pager should kick in? > > Yes, that could be it. At least it fails here, too, if I set > PAGER=less. Try: > > PAGER=more make dcheck > Surely for automated use you want the psql pager off altogether. "psql --pset pager=off" or some such invocation should do it. cheers andrew
Markus Wanner <markus@bluegap.ch> wrote: > So, the solution probably lies in adjusting the environment, > before starting psql. (Maybe even dropping all existing > environment variables for better control of the situation). Will > add that for dtester 0.1. Based on Andrew's suggestion, I changed line 276 to: args=['psql', '-A', '--pset=pager=off', I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds. I'm not clear on what you want to see from the run or whether it might be better sent off-list. Also, in looking closer at how you have the tests defined, it doesn't look to me like you're carefully interleaving specific sequences of statements on specific connections so much as opening multiple connections and then for each statement saying "run this on all connections." That's certainly a valid test to include, but I need the more controlled format, too. It does appear that that's pretty straightforward to code; you just haven't chosen to do so in the particular tests here, correct? -Kevin
"Kevin Grittner" <Kevin.Grittner@wicourts.gov> wrote: > Also, in looking closer at how you have the tests defined, it > doesn't look to me like you're carefully interleaving specific > sequences of statements on specific connections so much as opening > multiple connections and then for each statement saying "run this > on all connections." I take it back; you've got both. I do want to expand the tests quite a bit -- do I work them all into this same file, or how would I proceed? I think I'll need about 20 more tests, but I don't want to get in the way of your work on the framework which runs them. -Kevin
Markus Wanner <markus@bluegap.ch> wrote: > Please recheck with the StreamReporter and try to grep the lines > starting with "[psql0]", "[psql1]" and "[psql2]". Dtester simply > logs all and any output of all 3rd party processes started. For me, all psql output seems to be [psql0]; no [psql1] or [psql2]. Bug? -Kevin
Markus Wanner <markus@bluegap.ch> wrote: > Go try it, read the code and simply ask, if you get stuck. I'll > try to come up with some more documentation and such... I'm a little unclear about the differences between "uses", "depends", and "onlyAfter". Here's what they *sound* like they mean, to me; although I don't think the code isn't entirely consistent with this interpretation. "uses" means that the referenced task has complimentary setUp and tearDown methods, and the dependent task may only run after a successful invocation of the referenced task's setUp method, and the referenced task will wait for completion of all dependent tasks before invoking tearDown. "depends" means that the tearDown method of the referenced task doesn't undo the work of its setUp, at least for purposes of the dependent task. The dependent task can only start after successful completion of the referenced class's work (*just* setUp, or all the way to tearDown?), but the referenced task doesn't need to wait for the dependent task. "onlyAfter" means that the dependent task must wait for completion of the referenced task, but doesn't care whether or not the referenced class completed successfully. How close am I? -Kevin
Hi,
Kevin Grittner wrote:
> I'm a little unclear about the differences between "uses",
> "depends", and "onlyAfter". Here's what they *sound* like they
> mean, to me; although I don't think the code isn't entirely
> consistent with this interpretation.
Wow, you are way ahead of me. I intended to write some documentation
about that, but...
I differentiate tests and test suites. Tests mainly have a run method,
while test suites have setUp and tearDown ones.
> "uses" means that the referenced task has complimentary setUp and
> tearDown methods, and the dependent task may only run after a
> successful invocation of the referenced task's setUp method, and the
> referenced task will wait for completion of all dependent tasks
> before invoking tearDown.
Absolutely correct (may I just copy that para for documentation)? ;-)
Two additional things: tests and test suites may have requirements (in
the form of interfaces). The used test suites are passed to the
dependent task and it may call the referenced tasks's methods, for
example to get the database directory or to run a certain SQL command.
Second, if the referenced task fails, any running dependent task is
getting aborted as well. That might be obvious, though.
> "depends" means that the tearDown method of the referenced task
> doesn't undo the work of its setUp, at least for purposes of the
> dependent task. The dependent task can only start after successful
> completion of the referenced class's work (*just* setUp, or all the
> way to tearDown?), but the referenced task doesn't need to wait for
> the dependent task.
Hm.. no, not quite. The fact that not all suites clean up after them has
nothing to do with how they are referenced ("uses" or "depends"). So
far, it's entirely up to the test suite. I dislike that, but it works.
(I've been thinking about some separate resource allocation handling and
what not, but..)
The only difference between "depends" and "uses" is the requirements
fulfilling. "uses" does that, while "depends" only adds the timing and
functional dependencies, but doesn't pass the referenced task as an
argument to the dependent task.
> "onlyAfter" means that the dependent task must wait for completion
> of the referenced task, but doesn't care whether or not the
> referenced class completed successfully.
That's how I think it *should* be. ATM "onlyAfter" requires successful
completion of the dependent task.
I'd like to change that to support "onlyAfter", "onlyAfterSuccessOf" and
"onlyAfterFailureOf". Plus "onlyBefore" for convenience.
This is all work in progress and I'm open to suggestions and requests.
Thank you for thinking through all of this. I'm sure you understand now,
why it's not a version 0.1, yet :-)
Regards
Markus
Hi, Kevin Grittner wrote: > Based on Andrew's suggestion, I changed line 276 to: > > args=['psql', '-A', '--pset=pager=off', That looks like a correct fix for psql, yes. Thanks for pointing that out Andrew. Other processes might be confused by (or at least act differently with) a PAGER env variable, so that still needs to be cleared in general. > I now get 5 of 6 tests succeeded (83.3%), processed in 18.5 seconds. That's perfect. The one test that fails is expected to fail (another thing dtester doesn't support, yet). The serialization code you write should finally make that test pass ;-) > I do want to expand the tests quite a bit -- do I work them all into> this same file, or how would I proceed? I thinkI'll need about 20> more tests, but I don't want to get in the way of your work on the> framework which runs them. Well, first of all, another piece of the missing manual: there are BaseTest and SyncTest classes. Those based on BaseTest runs within the event loop of the twisted framework, thus need to be written in the very same asynchronous fashion. Mostly calling async methods that return a Deferred object, on which you may addCallback() or addErrback(). See the fine twisted documentation, especially the part about "Low-Level Networking and Event Loop" here: http://twistedmatrix.com/documents/current/core/howto/index.html The SyncTest is based on BaseTest, but a new thread is created to run its run method, passing back its results to the main event loop when done. That allows you to call blocking methods without having to care about blocking the entire event loop. However, it makes interacting between the two models a bit complicated. To call an async function from a SyncTest, you need to call the syncCall method. The separate thread then waits for some callback in the main event loop. Both have their own set of caveats, IMO. I'm not sure about how to organize the tests and ongoing development of the framework. I've already broken the Postgres-R tests with dtester-0.0. Maybe we put up a git branch with the dtester patches included? So whenever I want to change the framework, I can check if and how it affects your tests. Regards Markus
Markus Wanner wrote: >> I do want to expand the tests quite a bit -- do I work them all into >> this same file, or how would I proceed? I think I'll need about 20 >> more tests, but I don't want to get in the way of your work on the >> framework which runs them. > > Well, first of all, another piece of the missing manual: there are > BaseTest and SyncTest classes. Those based on BaseTest runs within the > event loop of the twisted framework, thus need to be written in the very > same asynchronous fashion. Mostly calling async methods that return a > Deferred object, on which you may addCallback() or addErrback(). See the > fine twisted documentation, especially the part about "Low-Level > Networking and Event Loop" here: > > http://twistedmatrix.com/documents/current/core/howto/index.html > I'm not sure about how to organize the tests and ongoing development of > the framework. I've already broken the Postgres-R tests with dtester-0.0. Hi, sorry to butt in to the conversation, but I have spent some time wrapping/refining the concepts in dtester, and the results are here: http://git.wulczer.org/?p=twisted-psql.git;a=summary It reqires Twisted and has been tested on Python 2.5 (should work on 2.6, no idea about 3.0). The program you use to run it - trial - should come with your distro's Twisted packages. The tests don't start a server or anything, so you need to have a PG instance running. To try it: git clone git://wulczer.org/twisted-psql.git cd twisted-psql # this is important, or Python won't find the modules $EDITOR config.py # set the path to psql and connection details for PG trial test.test_serialization_error trial test.test_true_serialization Both tests should pass, the latter being marked as an expectedFailure. You can then look at test/*.py to see my (puny) attempt at having some abstraction layer over the asynchronocity of the tests. I borrowed the idea of wrapping a psql in a Twisted protocol and added a Deferred interface around it, which made it possible to run tests with trial: the Twisted unit testing framework. As a developer of a failry large Python system based on Twisted, that sports hundreds of trial-based tests, I very strongly recommend trial for asynchronous unit testing. It handles lots of boring details, is well maintained and Twisted itself is simply designed to do asynchronous programming. As an added bonus, the runnning and reporting infrastructure is already there, you just write the tests. My code is very rough and lacks good error reporting, for instance failed tests will probably result in a "test hung" and the need to Ctrl+C, but that can be easily improved. A thing that would help tremendously would be a real Twisted protocol that talks to PG on the protocol level, not by parsing psql output (which is very clumsy and error prone IMHO). I found one such project: http://www.jamwt.com/pgasync/ but it had some issues with committing (all my test programs were exiting before PG got the final COMMIT, which resulted in the impossibility to do anything) and it does too much things that Python PG drivers like to do (like declaring a CURSOR for each query, bleah). A good implementation would hugely improve the quality and robustness of any such testsuite. Cheers, Jan -- Jan Urbanski GPG key ID: E583D7D2 ouden estin
Markus Wanner wrote:
Kevin Grittner wrote:
> I differentiate tests and test suites. Tests mainly have a run
> method, while test suites have setUp and tearDown ones.
I hadn't caught on to that distinction yet. That should help.
>> "uses" means that the referenced task has complimentary setUp and
>> tearDown methods, and the dependent task may only run after a
>> successful invocation of the referenced task's setUp method, and
>> the referenced task will wait for completion of all dependent
>> tasks before invoking tearDown.
>
> Absolutely correct (may I just copy that para for documentation)?
> ;-)
Use any of my language that you like, but I'm not generally known for
my word-smithing ability, so use at your own risk. ;-)
> Two additional things: tests and test suites may have requirements
> (in the form of interfaces). The used test suites are passed to the
> dependent task and it may call the referenced tasks's methods, for
> example to get the database directory or to run a certain SQL
> command.
Makes sense.
> Second, if the referenced task fails, any running dependent task is
> getting aborted as well. That might be obvious, though.
I figured that, although it's good to have it confirmed.
>> "depends" means that the tearDown method of the referenced task
>> doesn't undo the work of its setUp, at least for purposes of the
>> dependent task. The dependent task can only start after successful
>> completion of the referenced class's work (*just* setUp, or all
>> the way to tearDown?), but the referenced task doesn't need to
>> wait for the dependent task.
>
> Hm.. no, not quite. The fact that not all suites clean up after
> them has nothing to do with how they are referenced ("uses" or
> "depends"). So far, it's entirely up to the test suite. I dislike
> that, but it works. (I've been thinking about some separate
> resource allocation handling and what not, but..)
>
> The only difference between "depends" and "uses" is the
> requirements fulfilling. "uses" does that, while "depends" only
> adds the timing and functional dependencies, but doesn't pass the
> referenced task as an argument to the dependent task.
OK, that accounts for most of the differences between what they
sounded like to me and what I saw in the code. That's workable, now
that I understand it.
>> "onlyAfter" means that the dependent task must wait for completion
>> of the referenced task, but doesn't care whether or not the
>> referenced class completed successfully.
>
> That's how I think it *should* be. ATM "onlyAfter" requires
> successful completion of the dependent task.
That accounts for the rest of the differences.
> I'd like to change that to support "onlyAfter",
> "onlyAfterSuccessOf" and "onlyAfterFailureOf". Plus "onlyBefore"
> for convenience.
onlyAfterSuccessOf would be the same as depends with an empty
tearDown method? So it would effectively be syntactic sugar, for
convenience?
An onlyBefore reference from a to b would be semantically identical
to an onlyAfter reference from b to a? If so, that one seems to me
like it would muddy the waters more than it would help.
> Thank you for thinking through all of this. I'm sure you understand
> now, why it's not a version 0.1, yet :-)
Thank you for putting it together! I was afraid I was going to have
to go off-task on serializable implementation to write something so I
could test it. I'm more than happy to help stabilize your tool
instead!
-Kevin
Markus Wanner wrote: Kevin Grittner wrote: >> args=['psql', '-A', '--pset=pager=off', > That looks like a correct fix for psql, yes. > Other processes might be confused by (or at least act differently > with) a PAGER env variable, so that still needs to be cleared in > general. I see your point. Even with a general solution, probably best to leave the pset there for psql, though. > [discussion of BaseTest vs SyncTest] > > Both have their own set of caveats, IMO. I'll look those over. Any caveats beyond what you already mentioned of which I should be particularly careful? > Maybe we put up a git branch with the dtester patches included? So > whenever I want to change the framework, I can check if and how it > affects your tests. I strongly encourage you to set that up on git.postgresql.org. If for some reason that's not practicable, I can give you write permissions to my repository there and set up a dtester branch for this. I've barely scratched the surface on git in the last few weeks, and already I'm a big fan. I was never convinced that subversion was an improvement over cvs -- they each had advantages over the other which seemed a wash for me -- but git takes everything to a whole new level. -Kevin
Hi, Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>: > I strongly encourage you to set that up on git.postgresql.org. I'm about to provide git repositories for Postgres-R anyway, so I've setup two projects on git.postgres-r.org: dtester: that's the driver/harness code postgres-dtests: a Postgres clone with the dtester patch applied - this is based on the Postgres git repository, so you can easily switch between Postgres branches. I'd like to clean postgres-dtests in the sense that all tests included there are expected to succeed on Postgres HEAD. Those (like the initial SSI ones) that are expected to fail should get marked as such in there. If you want to add SSI specific tests, which your are expecting to succeed on your branch, I'd recommend you create your own branch (or merge into your branch from postgres-dtests). Git makes that simple enough. > I've barely scratched the surface on git in the last few > weeks, and already I'm a big fan. I was never convinced that > subversion was an improvement over cvs -- they each had advantages > over the other which seemed a wash for me -- but git takes everything > to a whole new level. I agree, and would like to extend that to DVCSes in general. Having started with monotone, I'm used to a different level of convenience, especially WRT network exchange and merging. To be fair, I'm excited about how fast git is (especially WRT network exchange, where monotone just plain sucks). > I see your point. Even with a general solution, probably best to > leave the pset there for psql, though. Agreed, that's fixed in the new git repository. > I'll look those over. Any caveats beyond what you already mentioned > of which I should be particularly careful? Uh.. no, there's no difference other than that. It's a paradigm question. Ones like it that way, others the other. And then there are applications that are a better fit than others... Now, I tend towards the event based approach, because it basically relieves you from having to deal with concurrent threads and all their issues. You need to get a single ordering of events anyway, if you want to check ordering constraints. Regards Markus
Hi, Jan Urbański wrote: > sorry to butt in to the conversation, but I have spent some time > wrapping/refining the concepts in dtester, and the results are here: > > http://git.wulczer.org/?p=twisted-psql.git;a=summary That seems to cover the concurrent psql part of dtester. But I don't see how that's wrapping or refining dtester. > I borrowed the idea of wrapping a psql in a Twisted protocol and added a > Deferred interface around it, which made it possible to run tests with > trial: the Twisted unit testing framework. dtester works pretty much the same way, except that it doesn't use trial. But the Deferred interface is about the same. > As a developer of a failry large Python system based on Twisted, that > sports hundreds of trial-based tests, I very strongly recommend trial > for asynchronous unit testing. It handles lots of boring details, is > well maintained and Twisted itself is simply designed to do asynchronous > programming. As an added bonus, the runnning and reporting > infrastructure is already there, you just write the tests. I'm using trial for two other projects as well and I've started with it for Postgres. IMO trial is very good for unit tests, but fails horribly for anything that involves complex setups and watch-guarding of multiple processes. That's where dtester shines. > My code is very rough and lacks good error reporting, for instance > failed tests will probably result in a "test hung" and the need to > Ctrl+C, but that can be easily improved. Don't underestimate that. There are lots of sources of errors. Not having a sentinel over the postmaster itself, unit tests from trial simply fail to notice errors there. (And coding for Postgres, these are the ones you are interested in). Now combine all of that and error handling and (useful) reporting is *the* major challenge in automating testing. > A thing that would help > tremendously would be a real Twisted protocol that talks to PG on the > protocol level, not by parsing psql output (which is very clumsy and > error prone IMHO). I agree. (Well, now I do). > I found one such project: > http://www.jamwt.com/pgasync/ > but it had some issues with committing Yeah, I didn't like that one, either. Regards Markus
[off-list to avoid distracting from the 9.0 wrap-up effort] Markus Wanner <markus@bluegap.ch> wrote: > Quoting "Kevin Grittner" <Kevin.Grittner@wicourts.gov>: >> I strongly encourage you to set that up on git.postgresql.org. > > I'm about to provide git repositories for Postgres-R anyway, so > I've setup two projects on git.postgres-r.org: > > dtester: that's the driver/harness code > postgres-dtests: a Postgres clone with the dtester patch applied - > this is based on the Postgres git repository, so you can easily > switch between Postgres branches. I just got to the point of having what appears to be a working but poorly optimized version of serializable transactions, so it is critical that I create a good set of tests to confirm correct behavior and monitor for regressions as I optimize. I see that you've been working on dtester recently -- should I grab what you've got here, stick with 0.1, or do you want to package something? If I should pull from your git, any hints on the best git statements to merge that in are welcome, I'm still rather new to git, and I tend not to get things right on a first try without some hints. :-/ Thanks again for this tool! -Kevin
I wrote: > [off-list to avoid distracting from the 9.0 wrap-up effort] Arg. I really didn't mean that to go to the list. :-( -Kevin
Hi, Kevin Grittner wrote: > I just got to the point of having what appears to be a working but > poorly optimized version of serializable transactions, Cool. > so it is > critical that I create a good set of tests to confirm correct > behavior and monitor for regressions as I optimize. I see that > you've been working on dtester recently -- should I grab what you've > got here, stick with 0.1, or do you want to package something? I'm preparing for a 0.1 release, yes. However, I'm trying to keep up with the postgres-dtests branch, so that should continue to work with the latest dtester code. > If I > should pull from your git, any hints on the best git statements to > merge that in are welcome, I'm still rather new to git, and I tend > not to get things right on a first try without some hints. :-/ I'd recommend merging postgres-dtests into your branch, so you'll get all of the adjustments from there, if I change dtester itself. Once 0.1 is out, I'm *trying* to remain backwards compatible in future releases. However, that holds me from releasing 0.1 in the first place :-) Independent of versions and releases: if you build your tests on top of dtester, I'm more than willing to support you with that. If you keep your code in a git repository, I could even provide patches, in case I need (or want) to change the dtester interface. Regards Markus Wanner