Thread: Suppressing occasional failures in copy2 regression test
Every so often the buildfarm shows row-ordering differences in the copy2 test, for example http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=jaguar&dt=2009-06-13%2003:00:02 ("jaguar" seems particularly prone to this for some reason, but other members have shown it too.) I believe what is happening is that autovacuum chances to trigger on the table being used, allowing some of the updated rows to be placed in positions they're not normally placed in. There is a simple fix for that: change the table to be a temp table, thus preventing autovac from touching it. Any objections to doing that? regards, tom lane
Sorry for top-posting -- stupid apple mail client... I'm not sure about that. It seems like race conditions with autovacuum are a real potential bug that it would be nice to be testing for. Another solution would be adding an order by clause - effectively trading coverage of unordered raw scans for coverage of the vacuum races. Or a third option would be adding alternate outputs for each ordering we observe. I suspect there aren't that many for serial tests but I'm less confident of that for the parallel tests. -- Greg On 13 Jun 2009, at 17:27, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Every so often the buildfarm shows row-ordering differences in the > copy2 > test, for example > http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=jaguar&dt=2009-06-13%2003:00:02 > ("jaguar" seems particularly prone to this for some reason, but other > members have shown it too.) I believe what is happening is that > autovacuum chances to trigger on the table being used, allowing some > of > the updated rows to be placed in positions they're not normally placed > in. > > There is a simple fix for that: change the table to be a temp table, > thus preventing autovac from touching it. > > Any objections to doing that? > > regards, tom lane > > -- > Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-hackers
Greg Stark <greg.stark@enterprisedb.com> writes:
> I'm not sure about that. It seems like race conditions with autovacuum
> are a real potential bug that it would be nice to be testing for.
It's not a bug; it's a limitation of our testing framework that it sees
this as a failure. Serious testing for autovac race conditions would
indeed be interesting, but you're never going to get anything meaningful
in that direction out of the current framework.
> Another solution would be adding an order by clause - effectively
> trading coverage of unordered raw scans for coverage of the vacuum
> races.
And destroying one of the main points of the copy2 test, which is that
those triggers are supposed to fire in a specific order.
> Or a third option would be adding alternate outputs for each ordering
> we observe. I suspect there aren't that many for serial tests but I'm
> less confident of that for the parallel tests.
There are several variants already observed, I believe, and I have
little confidence that there aren't more. In any case, that's a kluge
not a solution, and it still degrades the ability of the test to cover
what it was designed to cover.
regards, tom lane
On Sat, Jun 13, 2009 at 2:48 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote: > Greg Stark <greg.stark@enterprisedb.com> writes: >> I'm not sure about that. It seems like race conditions with autovacuum >> are a real potential bug that it would be nice to be testing for. > > It's not a bug; it's a limitation of our testing framework that it sees > this as a failure. Serious testing for autovac race conditions would > indeed be interesting, but you're never going to get anything meaningful > in that direction out of the current framework. The elephant in the room here may be moving to some more flexible/powerful testing framework, but the difficulty will almost certainly be in agreeing what it should look like. The actual writing of said test framework will take some work too, but to some degree that's a SMOP. This tuple-ordering issue seems to be one that comes up over and over again, but in the short term, making it a TEMP table seems like a reasonable fix. ...Robert
On Sun, Jun 14, 2009 at 12:39 AM, Robert Haas <robertmhaas@gmail.com> wrote:
I am forwarding a mail perl script and a pair of sample files that I developed about an year ago. The forwarded mail text explains what the script is trying to do. A line beginning with '?' in the expected file is treated specially.
If a line begins with '?' then the rest of the line is treated as a regular expression which will be used to match the corresponding line from the actual output.
If '?' is immediately followed by the word 'unordered' all the lines till a line containing '?/unordered' are buffered and compared against corresponding lines from the result file ignoring the order of the result lines.
Although we at EnterpriseDB have resolved the issues by alternate files etc., and do not use this script, I think it might be useful for community regression tests.
Best regards,
On Sat, Jun 13, 2009 at 2:48 PM, Tom Lane<tgl@sss.pgh.pa.us> wrote:The elephant in the room here may be moving to some more
> Greg Stark <greg.stark@enterprisedb.com> writes:
>> I'm not sure about that. It seems like race conditions with autovacuum
>> are a real potential bug that it would be nice to be testing for.
>
> It's not a bug; it's a limitation of our testing framework that it sees
> this as a failure. Serious testing for autovac race conditions would
> indeed be interesting, but you're never going to get anything meaningful
> in that direction out of the current framework.
flexible/powerful testing framework, but the difficulty will almost
certainly be in agreeing what it should look like. The actual writing
of said test framework will take some work too, but to some degree
that's a SMOP.
This tuple-ordering issue seems to be one that comes up over and over
again, but in the short term, making it a TEMP table seems like a
reasonable fix.
I am forwarding a mail perl script and a pair of sample files that I developed about an year ago. The forwarded mail text explains what the script is trying to do. A line beginning with '?' in the expected file is treated specially.
If a line begins with '?' then the rest of the line is treated as a regular expression which will be used to match the corresponding line from the actual output.
If '?' is immediately followed by the word 'unordered' all the lines till a line containing '?/unordered' are buffered and compared against corresponding lines from the result file ignoring the order of the result lines.
Although we at EnterpriseDB have resolved the issues by alternate files etc., and do not use this script, I think it might be useful for community regression tests.
Best regards,
---------- Forwarded message ----------
From: Gurjeet Singh <gurjeet.singh@enterprisedb.com>
Date: Fri, Aug 8, 2008 at 1:45 AM
Subject: neurodiff: a new diff utility for our regression test suites
Hi All,
PFA a perl script that implements a new kind of comparison, that might help us in situations like we have encountered with differeing plan costs in the hints patch recently. This script implements two new kinds of comparisons:
i) Regular Expression (RE) based comparison, and
ii) Comparison of unordered group of lines.
The input for this script, just like regular diff, are two files, one expected output and one the actual output. The lines in the expected output file which are expected to have any kind of variability should start with a '?' character followed by an RE that line should match.
For example, if we wish to compare a line of EXPLAIN output, that has the cost component too, then it might look like:
? Index Scan using accounts_i1 on accounts \(cost=\d+\.\d+\.\.\d+\.\d+ rows=\d+ width=\d+\)
The above RE would help us match any line that matches the pattern, such as:
Index Scan using accounts_i1 on accounts (cost=0.00..8.28 rows=1 width=106)
or
Index Scan using accounts_i1 on accounts (cost=1000.9999..2000.20008 rows=10000 width=1000)
Apart from this, the SQL standard does not guarantee any order of results unless the query has an explicit ORDER BY clause. We often encounter cases in our result files where the output differs from the expected only in the order of the result. To bypass this effect, and to keep the 'diff' quiet, I have seen people invariably add an ORDER BY clause to the query, and modify the expected file accordingly. There is a remote possibility of the ORDER BY clause masking an issue/bug that would have otherwise shown up in the diffs or might have caused the crash.
Using this script we can put special markers in the expected output, that denote the boundaries of a set of lines, that are expected to be produced in an undefined order. The script would not complain unless there's an actual missing or extra line in the output.
Suppose that we have the following result-set to compare:
4 | JACK
5 | CATHY
2 | SCOTT
1 | KING
3 | MILLER
The expected file would look like this:
?unordered
1 | KING
2 | SCOTT
?\d \| MILLER
4 | JACK
5 | CATHY
?/unordered
This expected file will succeed for both the following variations of the result-sets too:
5 | CATHY
4 | JACK
3 | MILLER
2 | SCOTT
1 | KING
or
1 | KING
4 | JACK
3 | MILLER
2 | SCOTT
5 | CATHY
Also, as shown in the above example, the RE based matching works for the lines within the 'unordered' set too.
The beauty of this approach for testing pattern matches and unordered results is that we don't have to modify the test cases in any way, just need to make adjustments in the expected output files.
I am no perl guru, so I definitely see a lot of performance/semantic improvements possible (perl gurus, take a stab); and maybe thats the reason the script looks more like a C program than a whacky perl script full of ~!$^ and whatnot.
This script cannot identify hunks, like 'diff' can do; which means that even if a single line is missing, or if there an extra line somewhere in the result file, all the rest of the lines from both the files will show up in the diff. But I think we do not need the hunk identification as much as we need the features this script provides.
Some time ago I had attempted to implement these very features in diffutils (diff et al.), but gave up too early! And then Dave's mention two days ago about trying to remove MinGW dependencies and moving to perl prompted me to start afresh in perl, and it was amazingly simple in perl (but was time consuming as I am a complete newbie)!
Best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
Attachment
On Mon, Jun 15, 2009 at 07:41:29PM +0530, Gurjeet Singh wrote: > On Sun, Jun 14, 2009 at 12:39 AM, Robert Haas <robertmhaas@gmail.com> wrote: > > > > Greg Stark <greg.stark@enterprisedb.com> writes: > > >> I'm not sure about that. It seems like race conditions with autovacuum > > >> are a real potential bug that it would be nice to be testing for. > > > > > > It's not a bug; it's a limitation of our testing framework that it sees > > > this as a failure. Serious testing for autovac race conditions would > > > indeed be interesting, but you're never going to get anything meaningful > > > in that direction out of the current framework. > > > > The elephant in the room here may be moving to some more > > flexible/powerful testing framework, but the difficulty will almost > > certainly be in agreeing what it should look like. The actual writing > > of said test framework will take some work too, but to some degree > > that's a SMOP. > > > > This tuple-ordering issue seems to be one that comes up over and over > > again, but in the short term, making it a TEMP table seems like a > > reasonable fix. > > > > I am forwarding a mail perl script and a pair of sample files that I > developed about an year ago. The forwarded mail text explains what the > script is trying to do. A line beginning with '?' in the expected file is > treated specially. > > If a line begins with '?' then the rest of the line is treated as a regular > expression which will be used to match the corresponding line from the > actual output. > > If '?' is immediately followed by the word 'unordered' all the lines till a > line containing '?/unordered' are buffered and compared against > corresponding lines from the result file ignoring the order of the result > lines. > > Although we at EnterpriseDB have resolved the issues by alternate files > etc., and do not use this script, I think it might be useful for community > regression tests. I looked over the program. Here are a few observations: * use strict; Good! * Fails perlcritic <http://search.cpan.org/~elliotjs/Perl-Critic-1.098/bin/perlcritic> at level 5: Bad! * It's not using Tie::File <http://perldoc.perl.org/Tie/File.html>. Probably bad. * It's going to a lot of trouble to allow for the possibility of both unordered results and of duplicate lines. If we disallowduplicate lines in unordered result sets, we can get a big speed gain by using hash-based comparisons. * Thanks, Gurjeet! :) Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
David Fetter <david@fetter.org> writes:
> * It's going to a lot of trouble to allow for the possibility of both
> unordered results and of duplicate lines. If we disallow duplicate
> lines in unordered result sets, we can get a big speed gain by using
> hash-based comparisons.
Why not just sort the lines and compare?
regards, tom lane
On Mon, Jun 15, 2009 at 11:34:38AM -0400, Tom Lane wrote: > David Fetter <david@fetter.org> writes: > > * It's going to a lot of trouble to allow for the possibility of both > > unordered results and of duplicate lines. If we disallow duplicate > > lines in unordered result sets, we can get a big speed gain by using > > hash-based comparisons. > > Why not just sort the lines and compare? Good point :) Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Mon, Jun 15, 2009 at 10:00 PM, David Fetter <david@fetter.org> wrote:
Good point :)On Mon, Jun 15, 2009 at 11:34:38AM -0400, Tom Lane wrote:
> David Fetter <david@fetter.org> writes:
> > * It's going to a lot of trouble to allow for the possibility of both
> > unordered results and of duplicate lines. If we disallow duplicate
> > lines in unordered result sets, we can get a big speed gain by using
> > hash-based comparisons.
>
> Why not just sort the lines and compare?
Please find attached the updated script and test cases. Changes since last submission:
.) Script uses tabs for indentation.
.) Script almost passes perlcritic.com at 'stern' level.
.) Correct some RE matches, so that a ? mark only at the beginning of the line matches (^ anchor).
.) Employ hybrid approach to support RE in unordered set, and to get better performance:
If there's no RE line in an unordered group of lines then perform sort on both arrays and then compare. If there _is_ an RE line in unordered group of lines, then do the O(n^2) processing to eliminate common lines and then report on missing lines.
TODO:
.) Using Tie::File to make code a little cleaner.
I agree that the choice of the hybrid approach for Unordered Set comparison makes script too indented, and maybe a little hard on eyes, but it's pretty simple and I have tried to delineate the major sections with proper comments.
Best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
Attachment
About the comment in chomp: did you try to use different values of $/? -- Alvaro Herrera http://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc.
<div dir="ltr"><div class="gmail_quote">On Sat, Jun 20, 2009 at 2:06 AM, Alvaro Herrera <span dir="ltr"><<a href="mailto:alvherre@commandprompt.com"target="_blank">alvherre@commandprompt.com</a>></span> wrote:<br /><blockquoteclass="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:1ex;"> About the comment in chomp: did you try to use different values of $/?<br /><font color="#888888"><br/></font></blockquote></div><br />Nope.. I think my first mail in this thread declared me a noob in perl:). So I'd appreciate if someone could improve it perl-wise.<br /><br />Thanks and best regards,<br />-- <br />Lets callit Postgres<br /><br />EnterpriseDB <a href="http://www.enterprisedb.com" target="_blank">http://www.enterprisedb.com</a><br/><br />gurjeet[.singh]@EnterpriseDB.com<br />singh.gurjeet@{ gmail | hotmail| indiatimes | yahoo }.com<br />Mail sent from my BlackLaptop device<br /></div>
<div dir="ltr"><div class="gmail_quote">On Sat, Jun 20, 2009 at 2:06 AM, Alvaro Herrera <span dir="ltr"><<a href="mailto:alvherre@commandprompt.com"target="_blank">alvherre@commandprompt.com</a>></span> wrote:<br /><blockquoteclass="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left:1ex;"> About the comment in chomp: did you try to use different values of $/?<br /><font color="#888888"><br/></font></blockquote></div><br />Well, now that I have tried it, yes, setting $/ to '\r\n' does giveme what I expected. Both expected and result files should have the same kind of line endings though.<br /><br />I expectedchomp() to "Do The Right Thing" and eat away '\r\n' too, not just '\n'; my mistake.<br /><br />Anyway, changing $/in the script does not seem to be much good for our test framework. Making sure that the expected and result files havea Unix style line endings would better stand the test of time.<br /><br />Another limitation of this script I forgotto mention is that it barfs if the last line of the expected file is '?/unordered'. Adding it to TODO list.<br /><br/>Best regards,<br /> -- <br />Lets call it Postgres<br /><br />EnterpriseDB <a href="http://www.enterprisedb.com"target="_blank">http://www.enterprisedb.com</a><br /><br />gurjeet[.singh]@EnterpriseDB.com<br/>singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com<br /> Mail sent from myBlackLaptop device<br /></div>
On Sun, Jun 21, 2009 at 10:36:04PM +0530, Gurjeet Singh wrote: > On Sat, Jun 20, 2009 at 2:06 AM, Alvaro Herrera > <alvherre@commandprompt.com>wrote: > > > About the comment in chomp: did you try to use different values of $/? > > > > > Well, now that I have tried it, yes, setting $/ to '\r\n' does give me what > I expected. Both expected and result files should have the same kind of line > endings though. Why would \r\n be in our code base anyhow? Cheers, David. -- David Fetter <david@fetter.org> http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fetter@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate
On Sun, Jun 21, 2009 at 11:37 PM, David Fetter <david@fetter.org> wrote:
I am not implying or expecting that we have \r\n in our test suite.
Its just that maybe my mail client introduced \r\n while downloading the old attachments, or probably my quite capable Notepad++ converted these files from \n to \r\n! I am not too sure. But it sure cost me a few cycles to realize that the files I copied from my Windows host to my Fedora VM were just not the thing perl would like.
On Sun, Jun 21, 2009 at 10:36:04PM +0530, Gurjeet Singh wrote:Why would \r\n be in our code base anyhow?
> On Sat, Jun 20, 2009 at 2:06 AM, Alvaro Herrera
> <alvherre@commandprompt.com>wrote:
>
> > About the comment in chomp: did you try to use different values of $/?
> >
> >
> Well, now that I have tried it, yes, setting $/ to '\r\n' does give me what
> I expected. Both expected and result files should have the same kind of line
> endings though.
I am not implying or expecting that we have \r\n in our test suite.
Its just that maybe my mail client introduced \r\n while downloading the old attachments, or probably my quite capable Notepad++ converted these files from \n to \r\n! I am not too sure. But it sure cost me a few cycles to realize that the files I copied from my Windows host to my Fedora VM were just not the thing perl would like.
Best regards,
--
Lets call it Postgres
EnterpriseDB http://www.enterprisedb.com
gurjeet[.singh]@EnterpriseDB.com
singh.gurjeet@{ gmail | hotmail | indiatimes | yahoo }.com
Mail sent from my BlackLaptop device
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon, Jun 22, 2009 at 03:16:05AM +0530, Gurjeet Singh wrote: [...] > [...] But it sure cost me a few cycles to > realize that the files I copied from my Windows host to my Fedora VM were > just not the thing perl would like. Had it been a Windows-ish perl, it would have worked. Chomp uses whatever the current value of "line separator" (or "input record separator", to be more precise) is (that's $/). Under Unix it's just "\n". Alas, $/ can't be a regular expression. The canonical way to deal with that can be seen e.g. here[1]. [1] <http://perldoc.perl.org/perlport.html#Newlines> Regards - -- tomás -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFKPxIiBcgs9XrR2kYRAqCzAJ9uVOoVGrGKMygL6W1xPVC0X629UQCfdwFc i7tYm+afcuu8DVo4c+mdYIM= =0rp6 -----END PGP SIGNATURE-----