Thread: Regression tests
While applying the NT regression tests, I remember Tom Lane's comment that people are being much more picky about the regression results. In the old days, we could just say that they will have _expected_ errors, but now they want them to match exactly. Kind of funny, their standards are going up. -- Bruce Momjian | http://www.op.net/~candle pgman@candle.pha.pa.us | (610) 853-3000+ If your life is a hard drive, | 830 Blythe Avenue + Christ can be your backup. | Drexel Hill, Pennsylvania19026
Bruce Momjian wrote: > > While applying the NT regression tests, I remember Tom Lane's comment > that people are being much more picky about the regression results. In > the old days, we could just say that they will have _expected_ errors, > but now they want them to match exactly. Well, it is NICE if they match exactly, but not essential for me. I have been advising intel RPM dist users to _expect_ float8 and geometry failures -- and the occassional 'random' failure, of course. And the situation with RedHat 6.1's locales brings yet another mixed bag -- I advise intel RPM users to completely disable RedHat's locale support (by renaming /etc/sysconfig/i18n to something else and rebooting) for performing regression tests -- then they can restore locale. Sort order under RedHat 6.1's locale is _messed_up_. Thomas mentioned RedHat 6.1 Intel being an appropriate reference platform -- if that is the case, then RH 6.1 Intel needs exact matches. Of course, ANY platform can be the reference -- as long as _one_ is. If BSD/os 4 (or whatever rev you're currently running) were to be the reference, then the regression tests had better match your machine's run. My opinion is to select the reference platform that produces the least amount of failures for other known good platforms. Also, geometry wouldn't fail if the number of digits of precision was jacked down one.... :-). > Kind of funny, their standards are going up. Is it a case of standards going up, or standards going down? ;-) As far as PostgreSQL's performance, our standards are definitely going up -- there has never been a better PostgreSQL, all around, than the one that is in CURRENT, IMO (and I've used everything since 6.1.1). -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
At 09:17 AM 3/31/00 -0500, Bruce Momjian wrote: >While applying the NT regression tests, I remember Tom Lane's comment >that people are being much more picky about the regression results. In >the old days, we could just say that they will have _expected_ errors, >but now they want them to match exactly. > >Kind of funny, their standards are going up. Is this perhaps a result of a growing audience for Postgres? For instance, I dealt with one of our web toolkit "early achievers", new to AOLserver, new to Postgres, new to the toolkit - that's a lot of "new to's" for someone to deal with in parallel! He had problems with the regression tests - cockpit error, first go-around, later diminished to expected errors. He's hacker enough to have run the regression tests in the first place (rather than blindly assume his install went OK) and also to figure out that the geometry results were probably due simply to FP imprecision, but wanted to safety-blanket reassurance from myself (and Lamar Owen) that all was A-OK. Particularly after his first go-around of self-inflicted problems (the details of which I don't even remember at the moment, he figured them out himself). As PG gets more use, I would expect to see more, not fewer, intellegent newcomers who aren't steeped in PG lore (i.e. experience with old versions) who will be full of questions about any seeming abnormality. - Don Baccus, Portland OR <dhogaza@pacifier.com> Nature photos, on-line guides, Pacific Northwest Rare Bird Alert Serviceand other goodies at http://donb.photo.net.
Lamar Owen writes: > Thomas mentioned RedHat 6.1 Intel being an appropriate reference > platform -- if that is the case, then RH 6.1 Intel needs exact matches. > Of course, ANY platform can be the reference -- as long as _one_ is. Why do we need a reference platform? Just to show "See, all the tests pass on this machine.", that has exactly zero practical value. If at all I think whatever hub.org is running should be the reference. A platform with broken locale certainly shouldn't. -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden
> > Thomas mentioned RedHat 6.1 Intel being an appropriate reference
> > platform -- if that is the case, then RH 6.1 Intel needs exact matches.
> > Of course, ANY platform can be the reference -- as long as _one_ is.
> Why do we need a reference platform? Just to show "See, all the tests pass
> on this machine.", that has exactly zero practical value.
Sorry, I disagree. When we first started maintaining the regression
tests, they passed on *no* machine, but every machine had different
failures (much as happens now). We need to have one machine defined as
the standard to simplify the testing process and discussion, and until
now that machine has been mine. If this is an issue, then, at least
for the 7.0 release, that machine will continue to be the same one, a
Linux RH5.2 machine at my home.
Until someone else takes *complete* responsibility for the regression
tests, does so for some period of time, and makes sure that these
tests are run regularly, then I can't see the situation changing. But
it is certainly true that for the last while (6 months, 1 year?) it is
clear that there are several people with enough interest and
persistance to take on this responsibility if they would like.
RH6.1 vs some other platform is a minor issue. I'd be happy for the
reference machine have Mandrake 7.0, which I'm running on several
machines.
- Thomas
--
Thomas Lockhart lockhart@alumni.caltech.edu
South Pasadena, California