Thread: Postgres alpha testing docs and general test packs

Hi,

Are there any test guides/plans generated for alpha releases, or are
such things only distributed to other developers?  I've seen postings
which mention what the new features are, and links to documentation
and other postings as to what it can do, but no single page outlining
the changes together.

And are there any test packs which can be run against each release to
ensure everything still functions as normal?  What I mean is it would
run through individual tests, like performing an update, checking
whether the update has applied, and returning pass if successful, and
fail otherwise.  Such tests should be inherently massive to match the
feature set of PostgreSQL, but could be built up over time if it
doesn't already exist.  Would there be any value in such a thing, or
is this generally not really a problem that needs solving?

Obviously real-world testing is needed to see how it works in a
realistic scenario, so I'm not suggesting that's any less important.

Thanks

Thom

2009/10/28 Grzegorz Jaśkiewicz <gryzman@gmail.com>:
> have you seen that one:
> http://it.toolbox.com/blogs/database-soup/alpha2-is-out-and-we-need-you-to-test-it-35032?rss=1
> ?

That's partly why I was asking.  It mentions the areas where the
changes have occurred, but not necessarily the changes themselves.  An
example of this is hstore.  There are mentions of improvements and
issues being eliminated, but these haven't been specified, not even in
the documentation.  I'm not sure how to test whatever change has gone
in.  I could open 8.4 and 8.5 documentation for that same page and
flip between the two until I find a difference, but even if I do that
and find changes, I doubt that covers what the fixes are.  I'd want
scenarios that were problematic in 8.4 that are not so in 8.5.

Entirely new features are easier to deal with though.  I still would,
however, want something like a detailed version of Josh's post which
breaks down where the changes have occurred.  It seems quite scattered
and unclear at the moment.

Thom

On Wednesday 28 October 2009 6:46:13 am Thom Brown wrote:
> 2009/10/28 Grzegorz Jaśkiewicz <gryzman@gmail.com>:
> > have you seen that one:
> > http://it.toolbox.com/blogs/database-soup/alpha2-is-out-and-we-need-you-t
> >o-test-it-35032?rss=1 ?
>
> That's partly why I was asking.  It mentions the areas where the
> changes have occurred, but not necessarily the changes themselves.  An
> example of this is hstore.  There are mentions of improvements and
> issues being eliminated, but these haven't been specified, not even in
> the documentation.  I'm not sure how to test whatever change has gone
> in.  I could open 8.4 and 8.5 documentation for that same page and
> flip between the two until I find a difference, but even if I do that
> and find changes, I doubt that covers what the fixes are.  I'd want
> scenarios that were problematic in 8.4 that are not so in 8.5.
>
> Entirely new features are easier to deal with though.  I still would,
> however, want something like a detailed version of Josh's post which
> breaks down where the changes have occurred.  It seems quite scattered
> and unclear at the moment.
>
> Thom

http://developer.postgresql.org/pgdocs/postgres/release-8-5.html

--
Adrian Klaver
aklaver@comcast.net

2009/10/28 Adrian Klaver <aklaver@comcast.net>:
>> Entirely new features are easier to deal with though.  I still would,
>> however, want something like a detailed version of Josh's post which
>> breaks down where the changes have occurred.  It seems quite scattered
>> and unclear at the moment.
>>
>> Thom
>
> http://developer.postgresql.org/pgdocs/postgres/release-8-5.html

Thanks Adrian.  I just wasn't looking hard enough obviously :)  That
list still doesn't appear to be explicit enough though as we have
"Multiple improvements in contrib/hstore, including raising limits on
keys and values".  What exactly is meant by limit, what was this limit
before and what has it been raised to?

Similarly: "Fix encoding handling in binary input function of xml
type."  What was the problem before?

And: "Allow the collection of statistics on sequences".  How would
your average end-user see whether these statistics are being colelcted
on sequences?  And are these statistics actually used anywhere yet?

I'm not really asking for the answer to those questions.  I'm pointing
out that it isn't clear (at least to me) how to determine what exactly
has been fixed in order to test it.  This doesn't apply to everything
listed as some of it is quite clear, like "pg_dump/pg_restore --clean
now drops large objects."

Thanks

Thom

Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
> 2009/10/28 Adrian Klaver <aklaver@comcast.net>:
> >> Entirely new features are easier to deal with though.  I still would,
> >> however, want something like a detailed version of Josh's post which
> >> breaks down where the changes have occurred.  It seems quite scattered
> >> and unclear at the moment.
> >>
> >> Thom
> >
> > http://developer.postgresql.org/pgdocs/postgres/release-8-5.html
>
> Thanks Adrian.  I just wasn't looking hard enough obviously :)  That
> list still doesn't appear to be explicit enough though as we have
> "Multiple improvements in contrib/hstore, including raising limits on
> keys and values".  What exactly is meant by limit, what was this limit
> before and what has it been raised to?
>
> Similarly: "Fix encoding handling in binary input function of xml
> type."  What was the problem before?
>
> And: "Allow the collection of statistics on sequences".  How would
> your average end-user see whether these statistics are being colelcted
> on sequences?  And are these statistics actually used anywhere yet?
>
> I'm not really asking for the answer to those questions.  I'm pointing
> out that it isn't clear (at least to me) how to determine what exactly
> has been fixed in order to test it.  This doesn't apply to everything
> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
> now drops large objects."
>

You're completely right. But release notes never intended to be this. What you
need is more a "visual" tour, and I don't think anyone did write such a thing
for any PostgreSQL releases (but I may be proven wrong). I wrote something
like this in french for 8.2, 8.3, and 8.4. The last two were even published in
a french Linux magazine. I suppose other people from other countries do the
same. The advocacy group would do a good thing if it starts working on this
kind of document. I could probably work on this too.


--
Guillaume.
 http://www.postgresqlfr.org
 http://dalibo.com

Le mercredi 28 octobre 2009 à 23:30:01, Adrian Klaver a écrit :
> ----- "Guillaume Lelarge" <guillaume@lelarge.info> wrote:
> > Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
> > > Similarly: "Fix encoding handling in binary input function of xml
> > > type."  What was the problem before?
>
> See attached screen shot for one possible solution.
>

This solution aims at developers, not users. I mean, I can do this and I
already do. My customers won't.


--
Guillaume.
 http://www.postgresqlfr.org
 http://dalibo.com

----- "Guillaume Lelarge" <guillaume@lelarge.info> wrote:

> Le mercredi 28 octobre 2009 à 23:30:01, Adrian Klaver a écrit :
> > ----- "Guillaume Lelarge" <guillaume@lelarge.info> wrote:
> > > Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
> > > > Similarly: "Fix encoding handling in binary input function of
> xml
> > > > type."  What was the problem before?
> >
> > See attached screen shot for one possible solution.
> >
>
> This solution aims at developers, not users. I mean, I can do this and
> I
> already do. My customers won't.
>
>
> --
> Guillaume.
>  http://www.postgresqlfr.org
>  http://dalibo.com


True, but I took it Thom was looking for a way to find answers in the meantime. This sort of comes under:
'Give a man a fish feed him for a day, teach him to fish feed him for a lifetime' :)


Adrian Klaver
aklaver@comcast.net

2009/10/28 Adrian Klaver <aklaver@comcast.net>:
>
>
>
> ----- "Guillaume Lelarge" <guillaume@lelarge.info> wrote:
>
>> Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
>
>> >
>> > Similarly: "Fix encoding handling in binary input function of xml
>> > type."  What was the problem before?
>> >
>
> See attached screen shot for one possible solution.
>

In other words we need to scour the committers mailing list to hunt
for this information?  This is exactly my point.  Testing doesn't
appear to be well organised.  In my last place of work we had a set of
requirements, technical solution design and a test guide which
instructed testers on what areas need testing.  From these a test plan
was built to ensure that the requirements were met, and that the
technical solution was working as specified.  In addition to this they
performed regression testing in the affected areas to ensure
everything else still worked as expected and wasn't negatively
affected by the new changes.

All we have are a summary of changes.  We can find out all the
information if we do plenty of searching of mailing lists and
comparing old and new documentation, but obviously this can be
off-putting and is duplicated for everyone who wants to participate in
testing.

I'm suggesting that while this is technically sufficient, it might be
a better idea to provide a clear technical document of the changes
that have been committed.

Such documentation may also potentially be reused when the final
version is released for end-users to review for any changes they might
need to make to their existing code and queries to ensure they don't
break.

Obviously PostgreSQL has survived very well without this, but I would
expect this would help more users perform more testing.

Thom

Thom Brown escribió:

> Obviously PostgreSQL has survived very well without this, but I would
> expect this would help more users perform more testing.

Keep in mind alphas are new.  Last time around, we only released a test
version when we were going to go to beta.  And the alpha idea was
accepted only because it was said that it was going to be very light on
the developer team.

If anyone (you?) wants to step up and produce the document you request,
it'll probably be linked to.  But please do not request the current
development team to work on it, because most of them are overloaded
already (or have other reasons not to).

--
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Wednesday 28 October 2009 3:55:02 pm Thom Brown wrote:
> 2009/10/28 Adrian Klaver <aklaver@comcast.net>:
> > ----- "Guillaume Lelarge" <guillaume@lelarge.info> wrote:
> >> Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
> >> > Similarly: "Fix encoding handling in binary input function of xml
> >> > type."  What was the problem before?
> >
> > See attached screen shot for one possible solution.
>
> In other words we need to scour the committers mailing list to hunt
> for this information?  This is exactly my point.  Testing doesn't
> appear to be well organised.  In my last place of work we had a set of
> requirements, technical solution design and a test guide which
> instructed testers on what areas need testing.  From these a test plan
> was built to ensure that the requirements were met, and that the
> technical solution was working as specified.  In addition to this they
> performed regression testing in the affected areas to ensure
> everything else still worked as expected and wasn't negatively
> affected by the new changes.

On the database side this handled by 'make check' which runs a regression suite
against the source. So this would be the first thing to do to ensure that the
database is not affected by a regression when compiled in your particular
environment. As to your particular needs/application things are a bit more
involved as you mention below. Here to date it seems most people have scanned
the change list for items that affected them and then dug deeper to get the
particulars. One of the benefits/problems of Open Source ,some assembly
required.

>
> All we have are a summary of changes.  We can find out all the
> information if we do plenty of searching of mailing lists and
> comparing old and new documentation, but obviously this can be
> off-putting and is duplicated for everyone who wants to participate in
> testing.
>
> I'm suggesting that while this is technically sufficient, it might be
> a better idea to provide a clear technical document of the changes
> that have been committed.

This can be seen as an opportunity to participate in the project. I am sure
plenty of people would be grateful if you where to spearhead just such a
document :)

>
> Such documentation may also potentially be reused when the final
> version is released for end-users to review for any changes they might
> need to make to their existing code and queries to ensure they don't
> break.
>
> Obviously PostgreSQL has survived very well without this, but I would
> expect this would help more users perform more testing.
>
> Thom

As was mentioned in another post the whole Alpha release program is new, so it
is still in the learning curve stage. My experience with the Postgres project
is that most itches do get scratched. It just does not always happen as fast as
everybody would like.

--
Adrian Klaver
aklaver@comcast.net

2009/10/28 Alvaro Herrera <alvherre@commandprompt.com>:
>
> If anyone (you?) wants to step up and produce the document you request,
> it'll probably be linked to.  But please do not request the current
> development team to work on it, because most of them are overloaded
> already (or have other reasons not to).

I can understand that, and I wouldn't expect our valuable developers
to do all this work.  I was thinking more of someone (or maybe more
than 1 person) taking the role of test documenter.... yes, yes, I know
I should get involved myself.  I will look into putting something
together to meet my own proposals.  I'm not entirely what the result
should look like for this particular project, but I'll see if I can
come up with something.  At least I might feel a little useful. ;)

Thom

On Wed, 28 Oct 2009, Thom Brown wrote:

> All we have are a summary of changes.  We can find out all the
> information if we do plenty of searching of mailing lists and comparing
> old and new documentation, but obviously this can be off-putting and is
> duplicated for everyone who wants to participate in testing.

For the last release, we had some people who updated blogs etc. with usage
examples for many of the new major features.  That doesn't seem to be
happening as well for the 8.5 development.

In any case, the whole process is still being worked out.  I for example
and working on some instructions for doing performance regression testing
of the alpha releases.  There actually is a full regression test suite
that gets runs all the time on many platforms.  The point of the alphas is
actually for you to try *your* tests, not for everyone to test the same
thing.

There is another route to get information here that might be a bit easier
than directly looking up things in the mailing lists or commit logs.
Each alpha is being generated after a CommitFest period during which
patches are commited.  The web application organizing that process
provides one way to more easily find the relevant discussion leading up
that patch being applied, and many of those include better/more obvious
examples and documentation.  The current alpha2 is based on the results of
https://commitfest.postgresql.org/action/commitfest_view?id=3

--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD

Le jeudi 29 octobre 2009 à 02:13:51, Greg Smith a écrit :
> On Wed, 28 Oct 2009, Thom Brown wrote:
> > All we have are a summary of changes.  We can find out all the
> > information if we do plenty of searching of mailing lists and comparing
> > old and new documentation, but obviously this can be off-putting and is
> > duplicated for everyone who wants to participate in testing.
>
> For the last release, we had some people who updated blogs etc. with usage
> examples for many of the new major features.  That doesn't seem to be
> happening as well for the 8.5 development.
>

I was thinking about this too. Lack of time, I suppose. But anyways, some blog
notes are interesting for testers, but hard to find if you need to go on
different blogs. All that kind of informations need to be found quite easily,
and the better way to do this is to put all of them in the same place. Maybe
the wiki can be such a place.

> In any case, the whole process is still being worked out.  I for example
> and working on some instructions for doing performance regression testing
> of the alpha releases.  There actually is a full regression test suite
> that gets runs all the time on many platforms.  The point of the alphas is
> actually for you to try *your* tests, not for everyone to test the same
> thing.
>
> There is another route to get information here that might be a bit easier
> than directly looking up things in the mailing lists or commit logs.
> Each alpha is being generated after a CommitFest period during which
> patches are commited.  The web application organizing that process
> provides one way to more easily find the relevant discussion leading up
> that patch being applied, and many of those include better/more obvious
> examples and documentation.  The current alpha2 is based on the results of
> https://commitfest.postgresql.org/action/commitfest_view?id=3
>

I didn't think about this, but it seems to be a better way to build this kind
of document than browsing the commit mailing list. I'll try to write something
according to informations available in 2009-07
(https://commitfest.postgresql.org/action/commitfest_view?id=2) and 2009-09
(https://commitfest.postgresql.org/action/commitfest_view?id=3) commitfest.
I'll begin the work next monday.


--
Guillaume.
 http://www.postgresqlfr.org
 http://dalibo.com

On ons, 2009-10-28 at 14:13 +0000, Thom Brown wrote:
> > http://developer.postgresql.org/pgdocs/postgres/release-8-5.html
>
> Thanks Adrian.  I just wasn't looking hard enough obviously :)  That
> list still doesn't appear to be explicit enough though as we have
> "Multiple improvements in contrib/hstore, including raising limits on
> keys and values".  What exactly is meant by limit, what was this limit
> before and what has it been raised to?
>
> Similarly: "Fix encoding handling in binary input function of xml
> type."  What was the problem before?
>
> And: "Allow the collection of statistics on sequences".  How would
> your average end-user see whether these statistics are being colelcted
> on sequences?  And are these statistics actually used anywhere yet?

I agree some of the release note items could be written in a more useful
way.  But ultimately, we can't elaborate on every code fix in detail.
Concentrate on the new features and try them out.  They should be
documented.  If not, or you can't find the documentation, please report
that.

> I'm not really asking for the answer to those questions.  I'm pointing
> out that it isn't clear (at least to me) how to determine what exactly
> has been fixed in order to test it.  This doesn't apply to everything
> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
> now drops large objects."

You can be reasonably assured that the particular fixes have been tested
and work, unless they are explicitly documented otherwise.  We don't
necessarily need more eyeballs to, say, check that the binary input
function of the xml type has *really* been fixed.

One point of the alpha releases is to test whether nothing else has been
broken by the various fixes, new features, and refactorings.  And you
can check that by running your application on top of the new database
server.  It helps if you have a test suite for your application.  For
example, if the fix of the binary input function of the xml type breaks
your application because it had relied on some undocumented corner case,
now would be good time to find that out.

2009/11/1 Peter Eisentraut <peter_e@gmx.net>:
> On ons, 2009-10-28 at 14:13 +0000, Thom Brown wrote:
>> I'm not really asking for the answer to those questions.  I'm pointing
>> out that it isn't clear (at least to me) how to determine what exactly
>> has been fixed in order to test it.  This doesn't apply to everything
>> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
>> now drops large objects."
>
> You can be reasonably assured that the particular fixes have been tested
> and work, unless they are explicitly documented otherwise.  We don't
> necessarily need more eyeballs to, say, check that the binary input
> function of the xml type has *really* been fixed.

Fair enough. :)  It's my experience in testing that makes me want to
test particular things to death.

> One point of the alpha releases is to test whether nothing else has been
> broken by the various fixes, new features, and refactorings.  And you
> can check that by running your application on top of the new database
> server.  It helps if you have a test suite for your application.  For
> example, if the fix of the binary input function of the xml type breaks
> your application because it had relied on some undocumented corner case,
> now would be good time to find that out.

Yeah, I should realise this is just an alpha release really.
Unfortunately I'd imagine that most places using PostgreSQL in
production won't be participating in alpha testing at all as there's
no call for it.

No doubt I can find what I'm looking for, even if it's looking through
the commit log.  But I guess that's your point, I probably don't need
to do that.

At the moment Gentoo doesn't even want for format my primary partition
(some DRDY error), so I still haven't had the chance to give this a
go. :(

Thom