Thread: Re: pgreplay log file replayer released

Re: pgreplay log file replayer released

From
Greg Stark
Date:
On Wed, Mar 17, 2010 at 2:06 PM, Albe Laurenz <laurenz.albe@wien.gv.at> wrote:
> I announce the first release of pgreplay, version 0.9.0 (Beta).
>
> Project home page: http://pgreplay.projects.postgresql.org/
>
> pgreplay reads a PostgreSQL log file (*not* a WAL file),
> extracts the SQL statements and executes them in the same order
> and relative time against a PostgreSQL database cluster.

Do you have a multi-threaded model that tracks which transactions each
query belonged to and runs them concurrently like they were in the
original setup? That's what I've been looking for.


--
greg

Re: pgreplay log file replayer released

From
Dimitri Fontaine
Date:
Greg Stark <gsstark@mit.edu> writes:
> Do you have a multi-threaded model that tracks which transactions each
> query belonged to and runs them concurrently like they were in the
> original setup? That's what I've been looking for.

Tsung does that and has been doing it for… quite some time. It even
comes with a recorder which is a PostgreSQL proxy: connect it to your
server, connect your client to it, and let it record a session at a
time.

Then in the configuration you get to choose how many of each session you
want to mix, etc.

  http://tsung.erlang-projects.org/

Regards,
--
dim

My TODO has "write a Tsung blog entry (series?) and a tutorial", but
you'll have to wait until after extensions and some other things, or do
it yourself... sorry about that...

Re: pgreplay log file replayer released

From
"Albe Laurenz"
Date:
Dimitri Fontaine wrote:
> Greg Stark <gsstark@mit.edu> writes:
>> Do you have a multi-threaded model that tracks which transactions each
>> query belonged to and runs them concurrently like they were in the
>> original setup? That's what I've been looking for.
> 
> Tsung does that and has been doing it for… quite some time. It even
> comes with a recorder which is a PostgreSQL proxy: connect it to your
> server, connect your client to it, and let it record a session at a
> time.
> 
> Then in the configuration you get to choose how many of each 
> session you
> want to mix, etc.
> 
>   http://tsung.erlang-projects.org/

pgreplay is single-threaded, but uses asynchronous query processing,
so multiple connections can be handled simultaneously.

pgreplay will use as many connections as the original run did, and
query order and timing are retained.

This is the first time I hear of Tsung - it sounds like a good idea.

I guess it has some advantages over pgreplay; the biggest one that
I can see is that it will see things that are not logged, like
COPY data.

It seems that Tsung currently only supports "basic queries", but I
assume that this can be improved. One thing that Tsung, recording
queries as proxy, will never be able to handle are encrypted connections,
but I guess that's a minor problem.

On the usability side, Tsung will require that all clients are redirected
to the recording proxy, while pgreplay will only require that the logging
configuration settings on the server are changed. This can be an advantage
in large distributed production environments.

Yours,
Laurenz Albe

Re: pgreplay log file replayer released

From
Dimitri Fontaine
Date:
"Albe Laurenz" <laurenz.albe@wien.gv.at> writes:
> It seems that Tsung currently only supports "basic queries", but I
> assume that this can be improved.

In fact from the time when PostgreSQL support was added, some more
Erlang drivers have appeared and some of them covers the entire
protocol. So it should be possible to update Tsung to use them, given
some interest.

> One thing that Tsung, recording
> queries as proxy, will never be able to handle are encrypted connections,
> but I guess that's a minor problem.

Yes, because you typically run the proxy only to record sessions, in
order to prepare the tsung setup. Another way to go from logs is to use
pgfouine, which knows how to prepare a tsung config from PostgreSQL
logs:

  http://pgfouine.projects.postgresql.org/tsung.html

> On the usability side, Tsung will require that all clients are redirected
> to the recording proxy, while pgreplay will only require that the logging
> configuration settings on the server are changed. This can be an advantage
> in large distributed production environments.

Well never use the tsung recorder in production. Ever. Run it to
construct your sessions files from your application.

Regards,
--
dim

Re: pgreplay log file replayer released

From
"Albe Laurenz"
Date:
Dimitri Fontaine wrote:
> > One thing that Tsung, recording
> > queries as proxy, will never be able to handle are encrypted connections,
> > but I guess that's a minor problem.
>
> Yes, because you typically run the proxy only to record sessions, in
> order to prepare the tsung setup. Another way to go from logs is to use
> pgfouine, which knows how to prepare a tsung config from PostgreSQL
> logs:
>
>   http://pgfouine.projects.postgresql.org/tsung.html

That's nice!

> > On the usability side, Tsung will require that all clients are redirected
> > to the recording proxy, while pgreplay will only require that the logging
> > configuration settings on the server are changed. This can be an advantage
> > in large distributed production environments.
>
> Well never use the tsung recorder in production. Ever. Run it to
> construct your sessions files from your application.

I see. So that case covers the creation of "artificial" session data as
opposed to the above case.

Yours,
Laurenz Albe

[SPAM] Re: pgreplay log file replayer released

From
Ben Chobot
Date:
On Mar 23, 2010, at 4:08 AM, Dimitri Fontaine wrote:

One thing that Tsung, recording
queries as proxy, will never be able to handle are encrypted connections,
but I guess that's a minor problem.

Yes, because you typically run the proxy only to record sessions, in
order to prepare the tsung setup. Another way to go from logs is to use
pgfouine, which knows how to prepare a tsung config from PostgreSQL
logs:

 http://pgfouine.projects.postgresql.org/tsung.html

We've actually found that pgfouine, while a great tool for most things, does not do a terribly good job at recreating production sessions for tsung. This isn't pgfouine's fault (although there is a bug there I still need to file) but rather with the fundamental design of Tsung and its origins as a tool to test stateless servers. If I want to replay a log *exactly* as it happened, tsung cannot help me. 

I definitely look forward to using pgreplay the next time we need to compare hardware classes on known production load.

Re: pgreplay log file replayer released

From
Greg Smith
Date:
I just summarized some of the discussion on this thread and created a
wiki page that starts to cover each of the three tools now available for
this job:  http://wiki.postgresql.org/wiki/Statement_Playback

--
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com   www.2ndQuadrant.us


[ANN]VTD-XML 2.9

From
"Jimmy Zhang"
Date:
VTD-XML 2.9, the next generation XML Processing API for SOA and Cloud computing, has been released. Please visit  https://sourceforge.net/projects/vtd-xml/files/ to download the latest version.
  • Strict Conformance
    • VTD-XML now fully conforms to XML namespace 1.0 spec
  • Performance Improvement
    • Significantly improved parsing performance for small XML files
  • Expand Core VTD-XML API 
    • Adds getPrefixString(), and toNormalizedString2()
  • Cutting/Splitting
    • Adds getSiblingElementFragment() 
  • A number of bug fixes and code enhancement including:
    • Fixes a bug for reading very large XML documents on some platforms
    • Fixes a bug in parsing processing instruction
    • Fixes a bug in outputAndReparse()