Thread: BUG - broken "make check" if different options

As I was investing playing around with blocksize, I noticed that some test 
cases under "make check" vary depending on compilation parameters, as 
they:
 - do not order the result of queries, thus are not deterministic   [join, with]
 - output query plans which differ depending on some parameters   [updatable_views, union, select_views]

Thus "make check" fails.
  sh> ./configure --blocksize=4  sh> make  sh> make check   ...   5 of 144 tests failed.

Adding some "ORDER BY" should solve the first issue, but ISTM that testing 
the output of query plans is a lost case for determinism, so maybe these 
test cases should be ignored/skipped when parameters are different.

-- 
Fabien.

Fabien COELHO <coelho@cri.ensmp.fr> writes:
> As I was investing playing around with blocksize, I noticed that some test 
> cases under "make check" vary depending on compilation parameters, as 
> they:

There has never been any expectation that the regression tests would pass
exactly no matter what the environment.  If we tried to make them do so,
we'd end up restricting the scope of testing so much as to be nearly
useless.

IOW, this is not a bug.
        regards, tom lane

>> As I was investing playing around with blocksize, I noticed that some test
>> cases under "make check" vary depending on compilation parameters, as
>> they:
>
> There has never been any expectation that the regression tests would 
> pass exactly no matter what the environment.  If we tried to make them 
> do so, we'd end up restricting the scope of testing so much as to be 
> nearly useless.

Hmmm... ok, so this is a feature.

I would have thought that tests about functional results should always 
pass, no matter what the environment, provided the environment allows to 
run the test, so basicaly queries should return deterministic results, 
implying an ORDER BY when there is more than one row in the output. So for 
me this part looks more like a bug than a feature.

For tests about plans, this is less obvious. Maybe test settings should 
control the environment enough so as to warrant deterministic results 
(say, tell the planner to assume this and this and this, and what is your 
plan ?). That would also help to test plan decisions with more extreme 
hardware. However, this would probably imply a test infrastructure beyond 
what is currently available. So I would be more ok for that part as a 
feature.


Also, this means that changing the default block size is basically never 
tested, otherwise the buildfarm would be reder than it is.

-- 
Fabien.