Thread: Concurrency testing

Concurrency testing

From
David Fetter
Date:
Folks,

As we move forward, we run into increasingly complex situations under
the general rubric of concurrency.

What test frameworks are already out there that we can use in our
regression test suite?  If there aren't any, how might we build one?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
David Fetter
Date:
On Wed, Oct 07, 2009 at 09:17:52AM -0700, David Fetter wrote:
> Folks,
> 
> As we move forward, we run into increasingly complex situations under
> the general rubric of concurrency.
> 
> What test frameworks are already out there that we can use in our
> regression test suite?  If there aren't any, how might we build one?

Apparently, this wasn't as crystal clear as I thought.  Let's imagine
you want to test some odd combination of transactions:

T1: SET TRANSACTION ISOLATION SERIALIZABLE;
T1: SELECT...FOR UPDATE...;
T2: UPDATE...
T3: COPY...
T1: ...

At the moment, we have no way to test this, although with certain Perl
modules, it would be pretty trivial.

I seem to recall that there were some patches to get psql to help with
such things, but they didn't go in.  Time to revive them?

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
Alvaro Herrera
Date:
David Fetter wrote:

> I seem to recall that there were some patches to get psql to help with
> such things, but they didn't go in.  Time to revive them?

Yeah, the API they implemented wasn't ideal, so there was some
discussion that ended up with a specification everyone was happy with,
but then nobody got around to implementing it.  Feel free to whack that
patch and resubmit ...  See in the archives around here:
http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.


Re: Concurrency testing

From
"David E. Wheeler"
Date:
On Oct 7, 2009, at 9:53 AM, David Fetter wrote:

> At the moment, we have no way to test this, although with certain Perl
> modules, it would be pretty trivial.

No non-core modules necessary. Just use Test::More and file handles  
opened via pipes to two or more psql sessions.

Best,

David


Re: Concurrency testing

From
David Fetter
Date:
On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote:
> On Oct 7, 2009, at 9:53 AM, David Fetter wrote:
>
>> At the moment, we have no way to test this, although with certain
>> Perl modules, it would be pretty trivial.
>
> No non-core modules necessary. Just use Test::More and file handles
> opened via pipes to two or more psql sessions.

When did Test::More become core?  I believe we support back to Perl
5.6 :/

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
"David E. Wheeler"
Date:
On Oct 7, 2009, at 10:38 AM, David Fetter wrote:

> When did Test::More become core?  I believe we support back to Perl
> 5.6 :/

Module::CoreList says 5.006002, though I would have sworn it was added  
much earlier than that. You could always use Test.pm, I suppose, which  
has been in core since 5.00405.

Best,

David


Re: Concurrency testing

From
Greg Smith
Date:
On Wed, 7 Oct 2009, Alvaro Herrera wrote:

> Yeah, the API they implemented wasn't ideal, so there was some
> discussion that ended up with a specification everyone was happy with,
> but then nobody got around to implementing it.

I needed something like this and didn't implement those suggestions 
because I thought the whole idea didn't scale up enough.  That's close to 
the right API to allow more complicated regression tests in psql itself, 
but I doubted that would hit real complexity level needed to find the good 
concurrent bugs.

The stuff I've been building lately takes "how many sessions at once?" as 
an input, and the useful range on that (particularly as we wander further 
toward multi-core land) is much higher than you'd want to manage one at a 
time manually.  I'd rather see a "session simulator" program that picks 
among several types of behaviors and executes them, and then you can throw 
as many of those as you want into the mix.  You can do something like that 
with pgbench right now if you pass it multiple files, the tricky part is 
figuring out what regression output you should expect.

We had a quick session on more complicated PG testing at PGCon, and it 
seems many of us who looked at this from multiple perspectives have 
settled into the idea that the current regression tests should stay as 
they are, and the more complicated ones are going to need to be in Perl to 
keep the code complexity managable but still allow more complicated tests. 
Another example of a complicted test is something that tests 
pg_dump/pg_restore, which is also hard to shoehorn into pg_regress.  I 
personally would rather see progress made in that area than trying to hack 
increasingly difficult tests into the existing regression framework, 
particular given how changes there impact existing committer's daily 
workflow.  The complexity level of test you need to find the juicy 
concurrent issues is not one you're going to want to run every time you do 
"make check".

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD


Re: Concurrency testing

From
Alvaro Herrera
Date:
Greg Smith wrote:
> On Wed, 7 Oct 2009, Alvaro Herrera wrote:
> 
> >Yeah, the API they implemented wasn't ideal, so there was some
> >discussion that ended up with a specification everyone was happy with,
> >but then nobody got around to implementing it.
> 
> I needed something like this and didn't implement those suggestions
> because I thought the whole idea didn't scale up enough.  That's
> close to the right API to allow more complicated regression tests in
> psql itself, but I doubted that would hit real complexity level
> needed to find the good concurrent bugs.

I don't find this a compelling argument against concurrent psql.  Sure
there are things you can't do with it, but it doesn't mean it's not
useful.  Are we going to need further tools to find "the good concurrent
bugs"?  No doubt.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


Performance testing framework..

From
Michael Renner
Date:
David Fetter wrote:
> Folks,
> 
> As we move forward, we run into increasingly complex situations under
> the general rubric of concurrency.
> 
> What test frameworks are already out there that we can use in our
> regression test suite?  If there aren't any, how might we build one?

Not entirely on-topic, but I thought I'd chime in:

After seeing the results from the hungarian blogger whose name I can't 
find [1] and quite a bit of discussion in the german usergroup before 
that I wanted to see how hard it would be to actually implement a 
framework to automate performance test runs.

I mainly used it as a reason to have a look at Moose [2], but since it 
was quite entertaining so far and I don't see any roadblocks I think 
I'll follow through.


Design Targets:

*) OO Design so it's easily pluggable, extensible (& testable ?)
*) Standalone CLI mode as well as daemonized "benchmark client" (think 
buildfarm)
*) Automate all aspects of benchmark operation, starting with a clean 
slate and (optionally) returning to same
*) Extensible Benchmark system, currently thinking about:  *) Synthetic cpu/memory/disk benchmarks (for evaluating and
verifying
 
the performance baseline of a given system, probably sysbench?)  *) pgbench  *) Whatever is freely available in the
DBT/TPCenvironment  *) As discussed, custom benchmarks testing specific subsystems with 
 
high concurrency/throughput, would need a proper tool I presume


Currently it doesn't do much more than building branches and creating 
clusters, but I guess I'll have something rudimentarily usable by mid 
november.

I haven't thought about result aggregation & rendering/UI part of the 
whole thing so far, so if anyone has some ideas in that direction they'd 
be very much appreciated when the time has come.


If I duplicated someones efforts already I'm sorry, I didn't want to 
raise anybodys hopes before I was sure that I really want to do this ;).


best regards,
Michael

[1] http://suckit.blog.hu/2009/09/26/postgresql_history
[2] http://www.iinteractive.com/moose/


Re: Concurrency testing

From
Greg Smith
Date:
On Wed, 7 Oct 2009, Alvaro Herrera wrote:

> I don't find this a compelling argument against concurrent psql.  Sure
> there are things you can't do with it, but it doesn't mean it's not
> useful.  Are we going to need further tools to find "the good concurrent
> bugs"?  No doubt.

Don't get me wrong, I wasn't arguing against concurrent psql being useful. 
Certainly it is.  I was just suggesting that the scale of issues it can be 
useful for is still pretty limited, and that accordingly I found my time 
better spent working on a higher-level solution that didn't need C-psql 
anyway.  Whether C-psql is sufficient for what David had in mind I can't 
say.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD


Re: Concurrency testing

From
David Fetter
Date:
On Wed, Oct 07, 2009 at 08:06:50PM -0400, Greg Smith wrote:
> On Wed, 7 Oct 2009, Alvaro Herrera wrote:
>
>> I don't find this a compelling argument against concurrent psql.
>> Sure there are things you can't do with it, but it doesn't mean
>> it's not useful.  Are we going to need further tools to find "the
>> good concurrent bugs"?  No doubt.
>
> Don't get me wrong, I wasn't arguing against concurrent psql being
> useful. Certainly it is.  I was just suggesting that the scale of
> issues it can be useful for is still pretty limited, and that
> accordingly I found my time better spent working on a higher-level
> solution that didn't need C-psql anyway.  Whether C-psql is
> sufficient for what David had in mind I can't say.

The kind of stuff I have in mind is regression tests.  As with
coverage and Dtrace, they might not be the ones that get run by
default, but right now, we have no real way in the regression tests to
test concurrency issues at all, as far as I know, so it would be good
to have some way to catch this stuff.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
Jeff Janes
Date:
On Wed, Oct 7, 2009 at 10:38 AM, David Fetter <david@fetter.org> wrote:
> On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote:
>> On Oct 7, 2009, at 9:53 AM, David Fetter wrote:
>>
>>> At the moment, we have no way to test this, although with certain
>>> Perl modules, it would be pretty trivial.
>>
>> No non-core modules necessary. Jus use Test::More and file handles
>> opened via pipes to two or more psql sessions.

I'd much rather live without Test::More and use DBD::Pg, then have
Test::More but need to open pipes to psql to talk to the database,
rather than using DBI to do it.  But I guess we would need to worry
about whether we can make DBD::Pg work with the installation being
tested, rather than finding some other install.

>
> When did Test::More become core?  I believe we support back to Perl
> 5.6 :/

Do we need to restrict ourselves to core?  Developers already need
flex and bison, which aren't needed when installing from the tarball.
Couldn't we also have "make dev-check" that has higher requirements
than "make check" does, but does a more thorough job?

Jeff


Re: Performance testing framework..

From
Greg Smith
Date:
On Wed, 7 Oct 2009, Michael Renner wrote:

> I haven't thought about result aggregation & rendering/UI part of the whole 
> thing so far, so if anyone has some ideas in that direction they'd be very 
> much appreciated when the time has come.

What I did in pgbench-tools (now available at http://git.postgresql.org/ ) 
is just dump all the individual run results, tagged by the name of test 
and similar key information, into a database.  That lets people aggregate 
however they want using SQL, and cleanly separates out the data from the 
UI.  My current rendering approach just pulls data from there into 
columnar text files I feed into gnuplot, but you could easily render with 
pretty much anything else instead just by tweaking how you pull things 
from the database.

All of the dbt-X tools (also in that git repo) instead use simple columnar 
text files (and their memory resident equivalent) to hold everything, 
feeding the data through the appropriate Perl libraries to compute stats 
and render things.  Since it drives me crazy to manipulate tables of data 
that could go into a database in a scripting language, I never bought into 
that approach.  But it's completely viable too for a standalone tool.

I'm not aware of anyone working on the job management side of the 
"performance farm" yet, so I don't think you duplicated anybody else's 
work.  Mark Wong (who manages the dbt stuff) and I have both been doing 
work on the benchmark results collection and organization side of things, 
but basically one test at a time.  The specific piece I've been working on 
lately is spawning off system monitoring daemons to collect information 
during the test, I think I'm on my 3rd generation of trying to get a 
solution I'm happy with to that problem.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD



Re: Concurrency testing

From
"David E. Wheeler"
Date:
On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:

> I'd much rather live without Test::More and use DBD::Pg, then have
> Test::More but need to open pipes to psql to talk to the database,
> rather than using DBI to do it.  But I guess we would need to worry
> about whether we can make DBD::Pg work with the installation being
> tested, rather than finding some other install.

The test architecture depends on Perl, but not on the DBI. I don't  
think that Andrew wants to add any dependencies. Therefore we'd need  
to use file handles. That's not to say that we couldn't write a nice  
little interface for it such that the implementation could later change.

> Do we need to restrict ourselves to core?  Developers already need
> flex and bison, which aren't needed when installing from the tarball.
> Couldn't we also have "make dev-check" that has higher requirements
> than "make check" does, but does a more thorough job?

flex and bison are not Perl modules.

Best,

David



Re: Performance testing framework..

From
Michael Renner
Date:
Greg Smith wrote:
> On Wed, 7 Oct 2009, Michael Renner wrote:
> 
>> I haven't thought about result aggregation & rendering/UI part of the 
>> whole thing so far, so if anyone has some ideas in that direction 
>> they'd be very much appreciated when the time has come.
> 
> What I did in pgbench-tools (now available at http://git.postgresql.org/ 

Thanks, I'll look into it! I think the nicest solution would be 
something in the liking of Sun's Analytics [1] framework, especially 
when you've got large amounts of historic data to correlate.  But this 
is probably a completely different beast to implement, though nice 
Javascript rendering libraries become commonplace these days. We'll see...


> I'm not aware of anyone working on the job management side of the 
> "performance farm" yet, so I don't think you duplicated anybody else's 
> work.

Good to hear :)


> things, but basically one test at a time.  The specific piece I've been 
> working on lately is spawning off system monitoring daemons to collect 
> information during the test, I think I'm on my 3rd generation of trying 
> to get a solution I'm happy with to that problem.

Ahh, that'd be also nice to have to see how ressource consumption 
differs between different runs & branches. This is probably highly 
platform dependent though...


Michael

[1] http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/


Re: Concurrency testing

From
Robert Haas
Date:
On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david@kineticode.com> wrote:
>> Do we need to restrict ourselves to core?  Developers already need
>> flex and bison, which aren't needed when installing from the tarball.
>> Couldn't we also have "make dev-check" that has higher requirements
>> than "make check" does, but does a more thorough job?
>
> flex and bison are not Perl modules.

True, but so what?

I don't really see what's wrong with using Perl modules that are
likely to be installed most places and easy to obtain where not, if it
makes writing a test framework much easier.  But I also think that we
should not get bogged down on exactly which tools to use - it seems to
me the first thing is to find someone who is willing to do the work.
If someone makes an AWESOME test suite that uses a module which is a
little too adventurous, we can probably find a way of adjusting it
after the fact so as to remove the dependency (I fancy myself fairly
good at this sort of thing, where Perl is concerned).  But if we argue
about tools now, we're just going to discourage anyone from taking a
stab at it.

Just MHO, of course.

...Robert


Re: Concurrency testing

From
Andrew Dunstan
Date:

David E. Wheeler wrote:
> On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:
>
>> I'd much rather live without Test::More and use DBD::Pg, then have
>> Test::More but need to open pipes to psql to talk to the database,
>> rather than using DBI to do it.  But I guess we would need to worry
>> about whether we can make DBD::Pg work with the installation being
>> tested, rather than finding some other install.
>
> The test architecture depends on Perl, but not on the DBI. I don't 
> think that Andrew wants to add any dependencies. Therefore we'd need 
> to use file handles. That's not to say that we couldn't write a nice 
> little interface for it such that the implementation could later change.

Well, that's true of the buildfarm. And there are reasons I don't want 
to use DBI for the buildfarm, mainly to do with its intended role in 
simulating what a human would do by hand.

What we do for the core testing framework is a different question. 
Nevertheless, a requirement for DBI and DBD::Pg would be a significant 
escalation of testing prerequisites. Test::More is comparatively modest 
requirement, and is fairly universal where Perl is installed. And since 
we'd just be using it to drive psql, we wouldn't be having to decide if 
a problem we saw was due to a problem in Postgres or a problem in DBD::Pg.

If we want something to drive a huge number of clients, I suspect 
neither of these is a good way to go, and something more custom built 
would be required. Last time I built something to drive a huge client 
load (many thousands of simultaneous connections to a web app) I did it 
in highly threaded Java using HttpUnit from a number of separate client 
machines. You wouldn't believe what that managed to do to MySQL on the 
backend ;-)


cheers

andrew



Re: Concurrency testing

From
Jeff Janes
Date:
On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas <robertmhaas@gmail.com> wrote:
> On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david@kineticode.com> wrote:
>>> Do we need to restrict ourselves to core?  Developers already need
>>> flex and bison, which aren't needed when installing from the tarball.
>>> Couldn't we also have "make dev-check" that has higher requirements
>>> than "make check" does, but does a more thorough job?
>>
>> flex and bison are not Perl modules.
>
> True, but so what?

Right, my point was only that we already have different levels of
requirements for different things.

>
> I don't really see what's wrong with using Perl modules that are
> likely to be installed most places and easy to obtain where not, if it
> makes writing a test framework much easier.  But I also think that we
> should not get bogged down on exactly which tools to use - it seems to
> me the first thing is to find someone who is willing to do the work.
> If someone makes an AWESOME test suite that uses a module which is a
> little too adventurous, we can probably find a way of adjusting it
> after the fact so as to remove the dependency (I fancy myself fairly
> good at this sort of thing, where Perl is concerned).  But if we argue
> about tools now, we're just going to discourage anyone from taking a
> stab at it.

OK, but what would we be taking a stab at?  Would it simply be
something like "make dev-check"  or "make concurrency-check", which
the build farm would then just invoke, thereby killing two birds with
one stone?  Or would it have to be something fancier than just that?
Or am I already too lost in the details?

Jeff


Re: Concurrency testing

From
Jeff Janes
Date:
On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler <david@kineticode.com> wrote:
> On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:
>
>> I'd much rather live without Test::More and use DBD::Pg, then have
>> Test::More but need to open pipes to psql to talk to the database,
>> rather than using DBI to do it.  But I guess we would need to worry
>> about whether we can make DBD::Pg work with the installation being
>> tested, rather than finding some other install.
>
> The test architecture depends on Perl, but not on the DBI. I don't think
> that Andrew wants to add any dependencies. Therefore we'd need to use file
> handles.

If we are going to talk to multiple psql pipes at the same time, and
not self-deadlock in the process, we would probably need to use some
form of non-blocking IO.  Unfortunately, I've often found it isn't as
easy as one might hope to do that in a completely portable way.  Maybe
IPC::Run, but I think that that is non-core also.

Are we interested only in simple deterministic concurrency testing: T1
does A, T2 does B, T1 does C and must block until T2 does D, at which
point (and not before) T1 must respond with E.  (Of course from some
recent testing I've done, we would also want "T1 does C and must block
either until T2 does D or until X+/-Y microseconds have passed")

Or would we want to beat the snot out the database in parallel in a
nondeterministic fashion, and checking for self-consistency and
non-deadlock?

Cheers,

Jeff


Re: Concurrency testing

From
Robert Haas
Date:
On Wed, Oct 7, 2009 at 10:41 PM, Jeff Janes <jeff.janes@gmail.com> wrote:
>> I don't really see what's wrong with using Perl modules that are
>> likely to be installed most places and easy to obtain where not, if it
>> makes writing a test framework much easier.  But I also think that we
>> should not get bogged down on exactly which tools to use - it seems to
>> me the first thing is to find someone who is willing to do the work.
>> If someone makes an AWESOME test suite that uses a module which is a
>> little too adventurous, we can probably find a way of adjusting it
>> after the fact so as to remove the dependency (I fancy myself fairly
>> good at this sort of thing, where Perl is concerned).  But if we argue
>> about tools now, we're just going to discourage anyone from taking a
>> stab at it.
>
> OK, but what would we be taking a stab at?  Would it simply be
> something like "make dev-check"  or "make concurrency-check", which
> the build farm would then just invoke, thereby killing two birds with
> one stone?  Or would it have to be something fancier than just that?
> Or am I already too lost in the details?

Well, I don't know.  I think the first thing would be to figure out
what sort of tests we want to run.  Taking a stab in the dark, I'm
thinking that the point of any sort of concurrency testing must be to
check for race conditions, so presumably it's the sort of thing you'd
want to have an easy way to run over and over again in a loop.
Possibly you would also want to test recovery by killing a backend in
the middle somewhere.  It may be too much to put into the core tree at
all; it might be easier to maintain as a separate project.  But I
don't have a strong opinion on this.  The subject of the thread is
"concurrency testing" so it seems to me we ought to start by figuring
out exactly what that means and how we could test it.

...Robert


Re: Concurrency testing

From
Dimitri Fontaine
Date:
Greg Smith <gsmith@gregsmith.com> writes: > The stuff I've been 
building lately takes "how many sessions at once?" as an > input, 
and the useful range on that (particularly as we wander further > 
toward multi-core land) is much higher than you'd want to manage 
one at a > time manually.  I'd rather see a "session simulator" 
program that picks > among several types of behaviors and executes 
them, and then you can throw > as many of those as you want into 
the mix.  You can do something like that > with pgbench right now 
if you pass it multiple files, the tricky part is > figuring out 
what regression output you should expect.   That tool exists and 
do support PostgreSQL. It's called Tsung and is Open Source:  http://tsung.erlang-projects.org/
http://tsung.erlang-projects.org/user_manual.htmlhttp://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php
 

It can handle thousands of connections per node and as many nodes 
as you
like, and already includes OS monitoring (adding to its own 
session /
transaction monitoring, of course). It also support SNMP here. Oh, 
and
thinktime too `randomize using a probability distribution 
(currently exponential)'.

I even started a DBT2 implementation as tsung modules, back when
returning from pgcon 2006:
 http://tapoueh.org/misc.html#sec7 darcs get http://pgsql.tapoueh.org/dbt2-tsung/

The thing that tsung isn't made for is tracking the results 
themselves
to validate against regressions. This would have to be done by 
other
means, maybe server side.

So Tsung ain't a direct answer to OP [waves to david], or only if 
we
want to add it the checking capability. I'll talk about it with 
tsung
main author. But it answers a lot of the content of this thread,
so I figured I'd better talk about it before it's being poorly
rewritten in perl.

Regards,
-- 
dim

PS: erlang is damn easy to learn.

PPS: sorry guys, perl is not made for concurrent programming, it's not
going to be easy reaching current tsung level (years of work with ad-hoc
tools) using a general hobbyist programming language.


Re: Concurrency testing

From
Dimitri Fontaine
Date:
Andrew Dunstan <andrew@dunslane.net> writes:
>  Last time I built something to drive a huge client load (many
> thousands of simultaneous connections to a web app) I did it in highly
> threaded Java using HttpUnit from a number of separate client machines. You
> wouldn't believe what that managed to do to MySQL on the backend ;-)

Last time I've head about that kind of Java stuff, it was a JMeter
story. The webapp guys and the JVM admins were so proud that the load
test couldn't worry their setup that they were willing to retest using
tsung.

On a single client rather than the client farm, tsung pushed their setup
to an halt under heavy load... and that was with older erlang limits of
about 800 connections per node, now it's on the 50000 connections per
node ballpark IIRC.

Just saying :)
-- 
dim


Re: Concurrency testing

From
David Fetter
Date:
On Thu, Oct 08, 2009 at 10:28:53AM +0200, Dimitri Fontaine wrote:
>  http://tsung.erlang-projects.org/
>  http://tsung.erlang-projects.org/user_manual.html
>  http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php

It looks like we're talking about two orthogonal projects here:

1.  Test situations which require more than one connection, possibly
to different clusters, for example in the cases of Hot Standby and
Streaming Replication.  This further divides into event-based and
time-based tests.  It's this situation I had in mind when I posted,
and it was inspired by bugs I've seen, the most recent being in
writeable CTEs.

2.  Test situations which require *many* connections in order to find
issues caused, in essence, by load.

Tsung seems well-suited to the latter.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
Greg Smith
Date:
On Thu, 8 Oct 2009, Dimitri Fontaine wrote:

> sorry guys, perl is not made for concurrent programming, it's not going 
> to be easy reaching current tsung level (years of work with ad-hoc 
> tools) using a general hobbyist programming language.

Perl is the best tool available that runs on the buildfarm, which makes 
something written in it useful here, while Tsung is not.  Unfortunately.

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD


Re: Concurrency testing

From
"Joshua D. Drake"
Date:
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote:
> On Thu, 8 Oct 2009, Dimitri Fontaine wrote:
>
> > sorry guys, perl is not made for concurrent programming, it's not going
> > to be easy reaching current tsung level (years of work with ad-hoc
> > tools) using a general hobbyist programming language.

Woah... I am not much of a Perl fan but to call it a "general hobbyist
programming language" certainly identifies the ignorance of the poster.


Joshua D. Drake

>
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
>
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

Re: Concurrency testing

From
Dimitri Fontaine
Date:
Le 8 oct. 2009 à 19:00, Greg Smith <gsmith@gregsmith.com> a écrit :
> Perl is the best tool available that runs on the buildfarm, which
> makes something written in it useful here, while Tsung is not.
> Unfortunately.

I though we were talking about 2 separate issues here: david's one,
for concurrent regression testing, and yours, for load stressing.

It does not look like the buildfarm is a good match for the later, so
I imagined we could see about bypassing perl this time ( given a
serious motive to ).

Regards,
--
dim
>


Re: Concurrency testing

From
Peter Eisentraut
Date:
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote:
> Perl is the best tool available that runs on the buildfarm, which makes 
> something written in it useful here, while Tsung is not.  Unfortunately.

But you're not going to do performance testing on the buildfarm anyway.



Re: Concurrency testing

From
"Joshua D. Drake"
Date:
On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote:
> On Thu, 8 Oct 2009, Dimitri Fontaine wrote:
> 
> > sorry guys, perl is not made for concurrent programming, it's not going 
> > to be easy reaching current tsung level (years of work with ad-hoc 
> > tools) using a general hobbyist programming language.

Woah... I am not much of a Perl fan but to call it a "general hobbyist
programming language" certainly identifies the ignorance of the poster.


Joshua D. Drake

> 
> --
> * Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
> 
-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander



Re: Concurrency testing

From
Markus Wanner
Date:
Hi,

David Fetter wrote:
> 1.  Test situations which require more than one connection, possibly
> to different clusters, for example in the cases of Hot Standby and
> Streaming Replication.  This further divides into event-based and
> time-based tests.  It's this situation I had in mind when I posted,
> and it was inspired by bugs I've seen, the most recent being in
> writeable CTEs.

Hm.. not sure what you mean by time-base tests, but to me that 
distinction sounds like regression vs. performance testing. Or do you 
think of time-based regression tests? I certainly favor regression tests 
that are event-based, as I've run into too many false errors due to 
unexpected timings already.

> 2.  Test situations which require *many* connections in order to find
> issues caused, in essence, by load.
> 
> Tsung seems well-suited to the latter.

Agreed. I would also note here, that Tsung seems especially well suited 
for performance testing of clustered solutions, as you very likely have 
to cluster the testing agents as well to put a decent load on the SUT.

Regards

Markus Wanner



Re: Concurrency testing

From
Markus Wanner
Date:
Hi,

Dimitri Fontaine wrote:
> I even started a DBT2 implementation as tsung modules, back when
> returning from pgcon 2006:
> 
>  http://tapoueh.org/misc.html#sec7
>  darcs get http://pgsql.tapoueh.org/dbt2-tsung/

Now *that* is very cool! I definitely need to have a look at it.

Have you tried Sun Faban? It allows to manage test runs, compare test 
results and configurations and features a web interface. Pretty nice as 
well. But it's all Java, which I like even less than Perl.

However, as far as I know, neither of the two support downloading code 
from a repository and building automatically, before testing. While the 
buildfarm already does that (partly, testing single patches would be a 
nice to have, too).

Regards

Markus Wanner


Re: Concurrency testing

From
Andrew Dunstan
Date:

Markus Wanner wrote:
> Hi,
>
> Dimitri Fontaine wrote:
>> I even started a DBT2 implementation as tsung modules, back when
>> returning from pgcon 2006:
>>
>>  http://tapoueh.org/misc.html#sec7
>>  darcs get http://pgsql.tapoueh.org/dbt2-tsung/
>
> Now *that* is very cool! I definitely need to have a look at it.
>
> Have you tried Sun Faban? It allows to manage test runs, compare test 
> results and configurations and features a web interface. Pretty nice 
> as well. But it's all Java, which I like even less than Perl.
>
> However, as far as I know, neither of the two support downloading code 
> from a repository and building automatically, before testing. While 
> the buildfarm already does that (partly, testing single patches would 
> be a nice to have, too).
>
>

It does, sort of. It has a switch called --from-source which lets you 
specify a local source repository, which can be something you've applied 
a patch to.

Use of this switch disables fetching the source code, checking it for 
cleanliness, and uploading the results to the central server.

It was designed for testing patches.

Here's how I use it:
   pushd srcroot/HEAD   cp -a pgsql pgsql.thistest   cd pgsql.thistest   cvs update   patch -p 0 <
/path/to/patch_to_test  popd   ./run_build.pl --verbose --from-source=`pwd`/srcroot/HEAD/pgsql.thistest
 

cheers

andrew


Re: Concurrency testing

From
David Fetter
Date:
On Fri, Oct 09, 2009 at 08:34:55PM +0200, Markus Schiltknecht wrote:
> Hi,
>
> David Fetter wrote:
>> 1.  Test situations which require more than one connection, possibly
>> to different clusters, for example in the cases of Hot Standby and
>> Streaming Replication.  This further divides into event-based and
>> time-based tests.  It's this situation I had in mind when I posted,
>> and it was inspired by bugs I've seen, the most recent being in
>> writeable CTEs.
>
> Hm.. not sure what you mean by time-base tests,

Kick off x.
Wait time t1
Kick off y.
Wait time t2.
etc., etc.

> but to me that  distinction sounds like regression vs. performance
> testing.  Or do you think of time-based regression tests? I
> certainly favor regression tests that are event-based, as I've run
> into too many false errors due to unexpected timings already.

Totally agree about regression tests.  There might be other kinds of
test--stress- or performance tests, for example--that would make more
sense time-based.

>> 2.  Test situations which require *many* connections in order to
>> find issues caused, in essence, by load.
>>
>> Tsung seems well-suited to the latter.
>
> Agreed. I would also note here, that Tsung seems especially well
> suited  for performance testing of clustered solutions, as you very
> likely have  to cluster the testing agents as well to put a decent
> load on the SUT.

I'd *love* to see PostgreSQL solutions that would outrun a single
decent Tsung box, but I suspect that's at least a year or two away.

Cheers,
David.
-- 
David Fetter <david@fetter.org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david.fetter@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


Re: Concurrency testing

From
Simon Riggs
Date:
On Wed, 2009-10-07 at 13:07 -0400, Alvaro Herrera wrote:
> David Fetter wrote:
> 
> > I seem to recall that there were some patches to get psql to help with
> > such things, but they didn't go in.  Time to revive them?
> 
> Yeah, the API they implemented wasn't ideal, so there was some
> discussion that ended up with a specification everyone was happy with,
> but then nobody got around to implementing it.  Feel free to whack that
> patch and resubmit ...  See in the archives around here:
> http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us

I would hope that Concurrent psql can be revived. There were some
issues, but not really major ones.

The main requirement is to be able to specify multiple sessions of
activity from a single script. I would prefer it if we could do that via
psql. If we start inventing new features in other tools we get situation
similar to pgbench, which has some cute features, but they aren't in
psql, which also has cute features, but different ones. Fragmentation
wastes effort.

I think Greg's comments are correct but I would say "also correct".
There is no reason to have just one test framework. We need as many as
we need.

-- Simon Riggs           www.2ndQuadrant.com