Thread: Concurrency testing
Folks, As we move forward, we run into increasingly complex situations under the general rubric of concurrency. What test frameworks are already out there that we can use in our regression test suite? If there aren't any, how might we build one? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Wed, Oct 07, 2009 at 09:17:52AM -0700, David Fetter wrote: > Folks, > > As we move forward, we run into increasingly complex situations under > the general rubric of concurrency. > > What test frameworks are already out there that we can use in our > regression test suite? If there aren't any, how might we build one? Apparently, this wasn't as crystal clear as I thought. Let's imagine you want to test some odd combination of transactions: T1: SET TRANSACTION ISOLATION SERIALIZABLE; T1: SELECT...FOR UPDATE...; T2: UPDATE... T3: COPY... T1: ... At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. I seem to recall that there were some patches to get psql to help with such things, but they didn't go in. Time to revive them? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter wrote: > I seem to recall that there were some patches to get psql to help with > such things, but they didn't go in. Time to revive them? Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. Feel free to whack that patch and resubmit ... See in the archives around here: http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
On Oct 7, 2009, at 9:53 AM, David Fetter wrote: > At the moment, we have no way to test this, although with certain Perl > modules, it would be pretty trivial. No non-core modules necessary. Just use Test::More and file handles opened via pipes to two or more psql sessions. Best, David
On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: > On Oct 7, 2009, at 9:53 AM, David Fetter wrote: > >> At the moment, we have no way to test this, although with certain >> Perl modules, it would be pretty trivial. > > No non-core modules necessary. Just use Test::More and file handles > opened via pipes to two or more psql sessions. When did Test::More become core? I believe we support back to Perl 5.6 :/ Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Oct 7, 2009, at 10:38 AM, David Fetter wrote: > When did Test::More become core? I believe we support back to Perl > 5.6 :/ Module::CoreList says 5.006002, though I would have sworn it was added much earlier than that. You could always use Test.pm, I suppose, which has been in core since 5.00405. Best, David
On Wed, 7 Oct 2009, Alvaro Herrera wrote: > Yeah, the API they implemented wasn't ideal, so there was some > discussion that ended up with a specification everyone was happy with, > but then nobody got around to implementing it. I needed something like this and didn't implement those suggestions because I thought the whole idea didn't scale up enough. That's close to the right API to allow more complicated regression tests in psql itself, but I doubted that would hit real complexity level needed to find the good concurrent bugs. The stuff I've been building lately takes "how many sessions at once?" as an input, and the useful range on that (particularly as we wander further toward multi-core land) is much higher than you'd want to manage one at a time manually. I'd rather see a "session simulator" program that picks among several types of behaviors and executes them, and then you can throw as many of those as you want into the mix. You can do something like that with pgbench right now if you pass it multiple files, the tricky part is figuring out what regression output you should expect. We had a quick session on more complicated PG testing at PGCon, and it seems many of us who looked at this from multiple perspectives have settled into the idea that the current regression tests should stay as they are, and the more complicated ones are going to need to be in Perl to keep the code complexity managable but still allow more complicated tests. Another example of a complicted test is something that tests pg_dump/pg_restore, which is also hard to shoehorn into pg_regress. I personally would rather see progress made in that area than trying to hack increasingly difficult tests into the existing regression framework, particular given how changes there impact existing committer's daily workflow. The complexity level of test you need to find the juicy concurrent issues is not one you're going to want to run every time you do "make check". -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
Greg Smith wrote: > On Wed, 7 Oct 2009, Alvaro Herrera wrote: > > >Yeah, the API they implemented wasn't ideal, so there was some > >discussion that ended up with a specification everyone was happy with, > >but then nobody got around to implementing it. > > I needed something like this and didn't implement those suggestions > because I thought the whole idea didn't scale up enough. That's > close to the right API to allow more complicated regression tests in > psql itself, but I doubted that would hit real complexity level > needed to find the good concurrent bugs. I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find "the good concurrent bugs"? No doubt. -- Alvaro Herrera http://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support
David Fetter wrote: > Folks, > > As we move forward, we run into increasingly complex situations under > the general rubric of concurrency. > > What test frameworks are already out there that we can use in our > regression test suite? If there aren't any, how might we build one? Not entirely on-topic, but I thought I'd chime in: After seeing the results from the hungarian blogger whose name I can't find [1] and quite a bit of discussion in the german usergroup before that I wanted to see how hard it would be to actually implement a framework to automate performance test runs. I mainly used it as a reason to have a look at Moose [2], but since it was quite entertaining so far and I don't see any roadblocks I think I'll follow through. Design Targets: *) OO Design so it's easily pluggable, extensible (& testable ?) *) Standalone CLI mode as well as daemonized "benchmark client" (think buildfarm) *) Automate all aspects of benchmark operation, starting with a clean slate and (optionally) returning to same *) Extensible Benchmark system, currently thinking about: *) Synthetic cpu/memory/disk benchmarks (for evaluating and verifying the performance baseline of a given system, probably sysbench?) *) pgbench *) Whatever is freely available in the DBT/TPCenvironment *) As discussed, custom benchmarks testing specific subsystems with high concurrency/throughput, would need a proper tool I presume Currently it doesn't do much more than building branches and creating clusters, but I guess I'll have something rudimentarily usable by mid november. I haven't thought about result aggregation & rendering/UI part of the whole thing so far, so if anyone has some ideas in that direction they'd be very much appreciated when the time has come. If I duplicated someones efforts already I'm sorry, I didn't want to raise anybodys hopes before I was sure that I really want to do this ;). best regards, Michael [1] http://suckit.blog.hu/2009/09/26/postgresql_history [2] http://www.iinteractive.com/moose/
On Wed, 7 Oct 2009, Alvaro Herrera wrote: > I don't find this a compelling argument against concurrent psql. Sure > there are things you can't do with it, but it doesn't mean it's not > useful. Are we going to need further tools to find "the good concurrent > bugs"? No doubt. Don't get me wrong, I wasn't arguing against concurrent psql being useful. Certainly it is. I was just suggesting that the scale of issues it can be useful for is still pretty limited, and that accordingly I found my time better spent working on a higher-level solution that didn't need C-psql anyway. Whether C-psql is sufficient for what David had in mind I can't say. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Wed, Oct 07, 2009 at 08:06:50PM -0400, Greg Smith wrote: > On Wed, 7 Oct 2009, Alvaro Herrera wrote: > >> I don't find this a compelling argument against concurrent psql. >> Sure there are things you can't do with it, but it doesn't mean >> it's not useful. Are we going to need further tools to find "the >> good concurrent bugs"? No doubt. > > Don't get me wrong, I wasn't arguing against concurrent psql being > useful. Certainly it is. I was just suggesting that the scale of > issues it can be useful for is still pretty limited, and that > accordingly I found my time better spent working on a higher-level > solution that didn't need C-psql anyway. Whether C-psql is > sufficient for what David had in mind I can't say. The kind of stuff I have in mind is regression tests. As with coverage and Dtrace, they might not be the ones that get run by default, but right now, we have no real way in the regression tests to test concurrency issues at all, as far as I know, so it would be good to have some way to catch this stuff. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Wed, Oct 7, 2009 at 10:38 AM, David Fetter <david@fetter.org> wrote: > On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: >> On Oct 7, 2009, at 9:53 AM, David Fetter wrote: >> >>> At the moment, we have no way to test this, although with certain >>> Perl modules, it would be pretty trivial. >> >> No non-core modules necessary. Jus use Test::More and file handles >> opened via pipes to two or more psql sessions. I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. > > When did Test::More become core? I believe we support back to Perl > 5.6 :/ Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have "make dev-check" that has higher requirements than "make check" does, but does a more thorough job? Jeff
On Wed, 7 Oct 2009, Michael Renner wrote: > I haven't thought about result aggregation & rendering/UI part of the whole > thing so far, so if anyone has some ideas in that direction they'd be very > much appreciated when the time has come. What I did in pgbench-tools (now available at http://git.postgresql.org/ ) is just dump all the individual run results, tagged by the name of test and similar key information, into a database. That lets people aggregate however they want using SQL, and cleanly separates out the data from the UI. My current rendering approach just pulls data from there into columnar text files I feed into gnuplot, but you could easily render with pretty much anything else instead just by tweaking how you pull things from the database. All of the dbt-X tools (also in that git repo) instead use simple columnar text files (and their memory resident equivalent) to hold everything, feeding the data through the appropriate Perl libraries to compute stats and render things. Since it drives me crazy to manipulate tables of data that could go into a database in a scripting language, I never bought into that approach. But it's completely viable too for a standalone tool. I'm not aware of anyone working on the job management side of the "performance farm" yet, so I don't think you duplicated anybody else's work. Mark Wong (who manages the dbt stuff) and I have both been doing work on the benchmark results collection and organization side of things, but basically one test at a time. The specific piece I've been working on lately is spawning off system monitoring daemons to collect information during the test, I think I'm on my 3rd generation of trying to get a solution I'm happy with to that problem. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: > I'd much rather live without Test::More and use DBD::Pg, then have > Test::More but need to open pipes to psql to talk to the database, > rather than using DBI to do it. But I guess we would need to worry > about whether we can make DBD::Pg work with the installation being > tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. That's not to say that we couldn't write a nice little interface for it such that the implementation could later change. > Do we need to restrict ourselves to core? Developers already need > flex and bison, which aren't needed when installing from the tarball. > Couldn't we also have "make dev-check" that has higher requirements > than "make check" does, but does a more thorough job? flex and bison are not Perl modules. Best, David
Greg Smith wrote: > On Wed, 7 Oct 2009, Michael Renner wrote: > >> I haven't thought about result aggregation & rendering/UI part of the >> whole thing so far, so if anyone has some ideas in that direction >> they'd be very much appreciated when the time has come. > > What I did in pgbench-tools (now available at http://git.postgresql.org/ Thanks, I'll look into it! I think the nicest solution would be something in the liking of Sun's Analytics [1] framework, especially when you've got large amounts of historic data to correlate. But this is probably a completely different beast to implement, though nice Javascript rendering libraries become commonplace these days. We'll see... > I'm not aware of anyone working on the job management side of the > "performance farm" yet, so I don't think you duplicated anybody else's > work. Good to hear :) > things, but basically one test at a time. The specific piece I've been > working on lately is spawning off system monitoring daemons to collect > information during the test, I think I'm on my 3rd generation of trying > to get a solution I'm happy with to that problem. Ahh, that'd be also nice to have to see how ressource consumption differs between different runs & branches. This is probably highly platform dependent though... Michael [1] http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/
On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david@kineticode.com> wrote: >> Do we need to restrict ourselves to core? Developers already need >> flex and bison, which aren't needed when installing from the tarball. >> Couldn't we also have "make dev-check" that has higher requirements >> than "make check" does, but does a more thorough job? > > flex and bison are not Perl modules. True, but so what? I don't really see what's wrong with using Perl modules that are likely to be installed most places and easy to obtain where not, if it makes writing a test framework much easier. But I also think that we should not get bogged down on exactly which tools to use - it seems to me the first thing is to find someone who is willing to do the work. If someone makes an AWESOME test suite that uses a module which is a little too adventurous, we can probably find a way of adjusting it after the fact so as to remove the dependency (I fancy myself fairly good at this sort of thing, where Perl is concerned). But if we argue about tools now, we're just going to discourage anyone from taking a stab at it. Just MHO, of course. ...Robert
David E. Wheeler wrote: > On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: > >> I'd much rather live without Test::More and use DBD::Pg, then have >> Test::More but need to open pipes to psql to talk to the database, >> rather than using DBI to do it. But I guess we would need to worry >> about whether we can make DBD::Pg work with the installation being >> tested, rather than finding some other install. > > The test architecture depends on Perl, but not on the DBI. I don't > think that Andrew wants to add any dependencies. Therefore we'd need > to use file handles. That's not to say that we couldn't write a nice > little interface for it such that the implementation could later change. Well, that's true of the buildfarm. And there are reasons I don't want to use DBI for the buildfarm, mainly to do with its intended role in simulating what a human would do by hand. What we do for the core testing framework is a different question. Nevertheless, a requirement for DBI and DBD::Pg would be a significant escalation of testing prerequisites. Test::More is comparatively modest requirement, and is fairly universal where Perl is installed. And since we'd just be using it to drive psql, we wouldn't be having to decide if a problem we saw was due to a problem in Postgres or a problem in DBD::Pg. If we want something to drive a huge number of clients, I suspect neither of these is a good way to go, and something more custom built would be required. Last time I built something to drive a huge client load (many thousands of simultaneous connections to a web app) I did it in highly threaded Java using HttpUnit from a number of separate client machines. You wouldn't believe what that managed to do to MySQL on the backend ;-) cheers andrew
On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas <robertmhaas@gmail.com> wrote: > On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david@kineticode.com> wrote: >>> Do we need to restrict ourselves to core? Developers already need >>> flex and bison, which aren't needed when installing from the tarball. >>> Couldn't we also have "make dev-check" that has higher requirements >>> than "make check" does, but does a more thorough job? >> >> flex and bison are not Perl modules. > > True, but so what? Right, my point was only that we already have different levels of requirements for different things. > > I don't really see what's wrong with using Perl modules that are > likely to be installed most places and easy to obtain where not, if it > makes writing a test framework much easier. But I also think that we > should not get bogged down on exactly which tools to use - it seems to > me the first thing is to find someone who is willing to do the work. > If someone makes an AWESOME test suite that uses a module which is a > little too adventurous, we can probably find a way of adjusting it > after the fact so as to remove the dependency (I fancy myself fairly > good at this sort of thing, where Perl is concerned). But if we argue > about tools now, we're just going to discourage anyone from taking a > stab at it. OK, but what would we be taking a stab at? Would it simply be something like "make dev-check" or "make concurrency-check", which the build farm would then just invoke, thereby killing two birds with one stone? Or would it have to be something fancier than just that? Or am I already too lost in the details? Jeff
On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler <david@kineticode.com> wrote: > On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: > >> I'd much rather live without Test::More and use DBD::Pg, then have >> Test::More but need to open pipes to psql to talk to the database, >> rather than using DBI to do it. But I guess we would need to worry >> about whether we can make DBD::Pg work with the installation being >> tested, rather than finding some other install. > > The test architecture depends on Perl, but not on the DBI. I don't think > that Andrew wants to add any dependencies. Therefore we'd need to use file > handles. If we are going to talk to multiple psql pipes at the same time, and not self-deadlock in the process, we would probably need to use some form of non-blocking IO. Unfortunately, I've often found it isn't as easy as one might hope to do that in a completely portable way. Maybe IPC::Run, but I think that that is non-core also. Are we interested only in simple deterministic concurrency testing: T1 does A, T2 does B, T1 does C and must block until T2 does D, at which point (and not before) T1 must respond with E. (Of course from some recent testing I've done, we would also want "T1 does C and must block either until T2 does D or until X+/-Y microseconds have passed") Or would we want to beat the snot out the database in parallel in a nondeterministic fashion, and checking for self-consistency and non-deadlock? Cheers, Jeff
On Wed, Oct 7, 2009 at 10:41 PM, Jeff Janes <jeff.janes@gmail.com> wrote: >> I don't really see what's wrong with using Perl modules that are >> likely to be installed most places and easy to obtain where not, if it >> makes writing a test framework much easier. But I also think that we >> should not get bogged down on exactly which tools to use - it seems to >> me the first thing is to find someone who is willing to do the work. >> If someone makes an AWESOME test suite that uses a module which is a >> little too adventurous, we can probably find a way of adjusting it >> after the fact so as to remove the dependency (I fancy myself fairly >> good at this sort of thing, where Perl is concerned). But if we argue >> about tools now, we're just going to discourage anyone from taking a >> stab at it. > > OK, but what would we be taking a stab at? Would it simply be > something like "make dev-check" or "make concurrency-check", which > the build farm would then just invoke, thereby killing two birds with > one stone? Or would it have to be something fancier than just that? > Or am I already too lost in the details? Well, I don't know. I think the first thing would be to figure out what sort of tests we want to run. Taking a stab in the dark, I'm thinking that the point of any sort of concurrency testing must be to check for race conditions, so presumably it's the sort of thing you'd want to have an easy way to run over and over again in a loop. Possibly you would also want to test recovery by killing a backend in the middle somewhere. It may be too much to put into the core tree at all; it might be easier to maintain as a separate project. But I don't have a strong opinion on this. The subject of the thread is "concurrency testing" so it seems to me we ought to start by figuring out exactly what that means and how we could test it. ...Robert
Greg Smith <gsmith@gregsmith.com> writes: > The stuff I've been building lately takes "how many sessions at once?" as an > input, and the useful range on that (particularly as we wander further > toward multi-core land) is much higher than you'd want to manage one at a > time manually. I'd rather see a "session simulator" program that picks > among several types of behaviors and executes them, and then you can throw > as many of those as you want into the mix. You can do something like that > with pgbench right now if you pass it multiple files, the tricky part is > figuring out what regression output you should expect. That tool exists and do support PostgreSQL. It's called Tsung and is Open Source: http://tsung.erlang-projects.org/ http://tsung.erlang-projects.org/user_manual.htmlhttp://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php It can handle thousands of connections per node and as many nodes as you like, and already includes OS monitoring (adding to its own session / transaction monitoring, of course). It also support SNMP here. Oh, and thinktime too `randomize using a probability distribution (currently exponential)'. I even started a DBT2 implementation as tsung modules, back when returning from pgcon 2006: http://tapoueh.org/misc.html#sec7 darcs get http://pgsql.tapoueh.org/dbt2-tsung/ The thing that tsung isn't made for is tracking the results themselves to validate against regressions. This would have to be done by other means, maybe server side. So Tsung ain't a direct answer to OP [waves to david], or only if we want to add it the checking capability. I'll talk about it with tsung main author. But it answers a lot of the content of this thread, so I figured I'd better talk about it before it's being poorly rewritten in perl. Regards, -- dim PS: erlang is damn easy to learn. PPS: sorry guys, perl is not made for concurrent programming, it's not going to be easy reaching current tsung level (years of work with ad-hoc tools) using a general hobbyist programming language.
Andrew Dunstan <andrew@dunslane.net> writes: > Last time I built something to drive a huge client load (many > thousands of simultaneous connections to a web app) I did it in highly > threaded Java using HttpUnit from a number of separate client machines. You > wouldn't believe what that managed to do to MySQL on the backend ;-) Last time I've head about that kind of Java stuff, it was a JMeter story. The webapp guys and the JVM admins were so proud that the load test couldn't worry their setup that they were willing to retest using tsung. On a single client rather than the client farm, tsung pushed their setup to an halt under heavy load... and that was with older erlang limits of about 800 connections per node, now it's on the 50000 connections per node ballpark IIRC. Just saying :) -- dim
On Thu, Oct 08, 2009 at 10:28:53AM +0200, Dimitri Fontaine wrote: > http://tsung.erlang-projects.org/ > http://tsung.erlang-projects.org/user_manual.html > http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php It looks like we're talking about two orthogonal projects here: 1. Test situations which require more than one connection, possibly to different clusters, for example in the cases of Hot Standby and Streaming Replication. This further divides into event-based and time-based tests. It's this situation I had in mind when I posted, and it was inspired by bugs I've seen, the most recent being in writeable CTEs. 2. Test situations which require *many* connections in order to find issues caused, in essence, by load. Tsung seems well-suited to the latter. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Thu, 8 Oct 2009, Dimitri Fontaine wrote: > sorry guys, perl is not made for concurrent programming, it's not going > to be easy reaching current tsung level (years of work with ad-hoc > tools) using a general hobbyist programming language. Perl is the best tool available that runs on the buildfarm, which makes something written in it useful here, while Tsung is not. Unfortunately. -- * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote: > On Thu, 8 Oct 2009, Dimitri Fontaine wrote: > > > sorry guys, perl is not made for concurrent programming, it's not going > > to be easy reaching current tsung level (years of work with ad-hoc > > tools) using a general hobbyist programming language. Woah... I am not much of a Perl fan but to call it a "general hobbyist programming language" certainly identifies the ignorance of the poster. Joshua D. Drake > > -- > * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
Le 8 oct. 2009 à 19:00, Greg Smith <gsmith@gregsmith.com> a écrit : > Perl is the best tool available that runs on the buildfarm, which > makes something written in it useful here, while Tsung is not. > Unfortunately. I though we were talking about 2 separate issues here: david's one, for concurrent regression testing, and yours, for load stressing. It does not look like the buildfarm is a good match for the later, so I imagined we could see about bypassing perl this time ( given a serious motive to ). Regards, -- dim >
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote: > Perl is the best tool available that runs on the buildfarm, which makes > something written in it useful here, while Tsung is not. Unfortunately. But you're not going to do performance testing on the buildfarm anyway.
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote: > On Thu, 8 Oct 2009, Dimitri Fontaine wrote: > > > sorry guys, perl is not made for concurrent programming, it's not going > > to be easy reaching current tsung level (years of work with ad-hoc > > tools) using a general hobbyist programming language. Woah... I am not much of a Perl fan but to call it a "general hobbyist programming language" certainly identifies the ignorance of the poster. Joshua D. Drake > > -- > * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander
Hi, David Fetter wrote: > 1. Test situations which require more than one connection, possibly > to different clusters, for example in the cases of Hot Standby and > Streaming Replication. This further divides into event-based and > time-based tests. It's this situation I had in mind when I posted, > and it was inspired by bugs I've seen, the most recent being in > writeable CTEs. Hm.. not sure what you mean by time-base tests, but to me that distinction sounds like regression vs. performance testing. Or do you think of time-based regression tests? I certainly favor regression tests that are event-based, as I've run into too many false errors due to unexpected timings already. > 2. Test situations which require *many* connections in order to find > issues caused, in essence, by load. > > Tsung seems well-suited to the latter. Agreed. I would also note here, that Tsung seems especially well suited for performance testing of clustered solutions, as you very likely have to cluster the testing agents as well to put a decent load on the SUT. Regards Markus Wanner
Hi, Dimitri Fontaine wrote: > I even started a DBT2 implementation as tsung modules, back when > returning from pgcon 2006: > > http://tapoueh.org/misc.html#sec7 > darcs get http://pgsql.tapoueh.org/dbt2-tsung/ Now *that* is very cool! I definitely need to have a look at it. Have you tried Sun Faban? It allows to manage test runs, compare test results and configurations and features a web interface. Pretty nice as well. But it's all Java, which I like even less than Perl. However, as far as I know, neither of the two support downloading code from a repository and building automatically, before testing. While the buildfarm already does that (partly, testing single patches would be a nice to have, too). Regards Markus Wanner
Markus Wanner wrote: > Hi, > > Dimitri Fontaine wrote: >> I even started a DBT2 implementation as tsung modules, back when >> returning from pgcon 2006: >> >> http://tapoueh.org/misc.html#sec7 >> darcs get http://pgsql.tapoueh.org/dbt2-tsung/ > > Now *that* is very cool! I definitely need to have a look at it. > > Have you tried Sun Faban? It allows to manage test runs, compare test > results and configurations and features a web interface. Pretty nice > as well. But it's all Java, which I like even less than Perl. > > However, as far as I know, neither of the two support downloading code > from a repository and building automatically, before testing. While > the buildfarm already does that (partly, testing single patches would > be a nice to have, too). > > It does, sort of. It has a switch called --from-source which lets you specify a local source repository, which can be something you've applied a patch to. Use of this switch disables fetching the source code, checking it for cleanliness, and uploading the results to the central server. It was designed for testing patches. Here's how I use it: pushd srcroot/HEAD cp -a pgsql pgsql.thistest cd pgsql.thistest cvs update patch -p 0 < /path/to/patch_to_test popd ./run_build.pl --verbose --from-source=`pwd`/srcroot/HEAD/pgsql.thistest cheers andrew
On Fri, Oct 09, 2009 at 08:34:55PM +0200, Markus Schiltknecht wrote: > Hi, > > David Fetter wrote: >> 1. Test situations which require more than one connection, possibly >> to different clusters, for example in the cases of Hot Standby and >> Streaming Replication. This further divides into event-based and >> time-based tests. It's this situation I had in mind when I posted, >> and it was inspired by bugs I've seen, the most recent being in >> writeable CTEs. > > Hm.. not sure what you mean by time-base tests, Kick off x. Wait time t1 Kick off y. Wait time t2. etc., etc. > but to me that distinction sounds like regression vs. performance > testing. Or do you think of time-based regression tests? I > certainly favor regression tests that are event-based, as I've run > into too many false errors due to unexpected timings already. Totally agree about regression tests. There might be other kinds of test--stress- or performance tests, for example--that would make more sense time-based. >> 2. Test situations which require *many* connections in order to >> find issues caused, in essence, by load. >> >> Tsung seems well-suited to the latter. > > Agreed. I would also note here, that Tsung seems especially well > suited for performance testing of clustered solutions, as you very > likely have to cluster the testing agents as well to put a decent > load on the SUT. I'd *love* to see PostgreSQL solutions that would outrun a single decent Tsung box, but I suspect that's at least a year or two away. Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Wed, 2009-10-07 at 13:07 -0400, Alvaro Herrera wrote: > David Fetter wrote: > > > I seem to recall that there were some patches to get psql to help with > > such things, but they didn't go in. Time to revive them? > > Yeah, the API they implemented wasn't ideal, so there was some > discussion that ended up with a specification everyone was happy with, > but then nobody got around to implementing it. Feel free to whack that > patch and resubmit ... See in the archives around here: > http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us I would hope that Concurrent psql can be revived. There were some issues, but not really major ones. The main requirement is to be able to specify multiple sessions of activity from a single script. I would prefer it if we could do that via psql. If we start inventing new features in other tools we get situation similar to pgbench, which has some cute features, but they aren't in psql, which also has cute features, but different ones. Fragmentation wastes effort. I think Greg's comments are correct but I would say "also correct". There is no reason to have just one test framework. We need as many as we need. -- Simon Riggs www.2ndQuadrant.com