Thread: mb and ecpg regression tests
We've had ecpg regression tests being doing by the buildfarm for a while, but they are not enabled for MSVC. Also, buildfarm has never supported doing MBCS regression tests. In both cases the regression tests are driven by Unix shell scripts, and in the MBCS case at least, the script would need plenty of love before it would be suitable for the buildfarm even on Unix. Back when we were thinking of supporting MSVC builds, we thought that everything would need to be a C program, and so we rewrote the main regression script in C, for example. However, now we have a build system for MSVC we can see that its main infrastructure is in fact perl, so I think what we need to do is to convert the MBCS and ECPG regression drivers to perl rather than C - which should be far, far simpler task. I'm going to try to get these done and supported by the buildfarm by the time we get to Beta. cheers andrew
Andrew Dunstan wrote: > > We've had ecpg regression tests being doing by the buildfarm for a > while, but they are not enabled for MSVC. Also, buildfarm has never > supported doing MBCS regression tests. In both cases the regression > tests are driven by Unix shell scripts, and in the MBCS case at least, > the script would need plenty of love before it would be suitable for the > buildfarm even on Unix. > > Back when we were thinking of supporting MSVC builds, we thought that > everything would need to be a C program, and so we rewrote the main > regression script in C, for example. However, now we have a build system > for MSVC we can see that its main infrastructure is in fact perl, so I > think what we need to do is to convert the MBCS and ECPG regression > drivers to perl rather than C - which should be far, far simpler task. > > I'm going to try to get these done and supported by the buildfarm by the > time we get to Beta. I don't think it's essential, but one of the other reasons Magnus and I discussed for rewriting the regression tests in C was that we could include it in the future with the installer as a platform/installation confidence test. Regards, Dave.
Dave Page wrote: > Andrew Dunstan wrote: > >> We've had ecpg regression tests being doing by the buildfarm for a >> while, but they are not enabled for MSVC. Also, buildfarm has never >> supported doing MBCS regression tests. In both cases the regression >> tests are driven by Unix shell scripts, and in the MBCS case at least, >> the script would need plenty of love before it would be suitable for the >> buildfarm even on Unix. >> >> Back when we were thinking of supporting MSVC builds, we thought that >> everything would need to be a C program, and so we rewrote the main >> regression script in C, for example. However, now we have a build system >> for MSVC we can see that its main infrastructure is in fact perl, so I >> think what we need to do is to convert the MBCS and ECPG regression >> drivers to perl rather than C - which should be far, far simpler task. >> >> I'm going to try to get these done and supported by the buildfarm by the >> time we get to Beta. >> > > I don't think it's essential, but one of the other reasons Magnus and I > discussed for rewriting the regression tests in C was that we could > include it in the future with the installer as a platform/installation > confidence test. > > > That's a good point. So, do we want/need the same for MBCS and ECPG tests? If so, we should work on creating a regression shared lib so we don't have to replicate lots of code. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> Back when we were thinking of supporting MSVC builds, we thought that
> everything would need to be a C program, and so we rewrote the main
> regression script in C, for example. However, now we have a build system
> for MSVC we can see that its main infrastructure is in fact perl, so I
> think what we need to do is to convert the MBCS and ECPG regression
> drivers to perl rather than C - which should be far, far simpler task.
I disagree with this for ecpg because (a) it would be a capability
regression from the point of view of anyone building a non-perl-enabled
build on Unix, and (b) we already have the C regression driver, why
can't we re-use or adapt it for ecpg instead of writing and debugging
and maintaining an entirely new set of test infrastructure?
For the MBCS stuff I don't care, since no one runs that on a regular
basis anyway (or should need to...)
regards, tom lane
Tom Lane wrote: > Andrew Dunstan <andrew@dunslane.net> writes: > >> Back when we were thinking of supporting MSVC builds, we thought that >> everything would need to be a C program, and so we rewrote the main >> regression script in C, for example. However, now we have a build system >> for MSVC we can see that its main infrastructure is in fact perl, so I >> think what we need to do is to convert the MBCS and ECPG regression >> drivers to perl rather than C - which should be far, far simpler task. >> > > I disagree with this for ecpg because (a) it would be a capability > regression from the point of view of anyone building a non-perl-enabled > build on Unix, and (b) we already have the C regression driver, why > can't we re-use or adapt it for ecpg instead of writing and debugging > and maintaining an entirely new set of test infrastructure? > > For the MBCS stuff I don't care, since no one runs that on a regular > basis anyway (or should need to...) > > > Well, you need perl to build from CVS, IIRC, but I take the point. I don't think ECPG can use pg_regress as is, because it does more than just run psql. We probably need to look at factoring out the common bits into a shared lib. That's not a bad idea anyway. cheers andrew
Andrew Dunstan <andrew@dunslane.net> writes:
> I don't think ECPG can use pg_regress as is, because it does more than
> just run psql. We probably need to look at factoring out the common bits
> into a shared lib. That's not a bad idea anyway.
A shared library is probably overkill (it has a lot more installation
overhead than it's worth), but at least factor the source
code so we don't have two copies of the common bits.
It wouldn't be a bad goal to try to use the same "pg_regress 2.0" for
both uses, either. I'm tempted to suggest it should become a separate
subdirectory under src/bin/ to remind people it's decoupled from the
regression tests per se...
regards, tom lane
Andrew Dunstan wrote: > > > That's a good point. So, do we want/need the same for MBCS and ECPG tests? It would be nice for ECPG. I wasn't even aware of the MBCS stuff - if it will exercise the unicode sorting which is a little different on Windows iirc (due to the lack of full native UTF-8 support) it might be a useful addition, but I wouldn't say it was worth busting a gut over. Regards, Dave.
Andrew Dunstan wrote: > > We've had ecpg regression tests being doing by the buildfarm for a > while, but they are not enabled for MSVC. Also, buildfarm has never > supported doing MBCS regression tests. In both cases the regression > tests are driven by Unix shell scripts, and in the MBCS case at least, > the script would need plenty of love before it would be suitable for the > buildfarm even on Unix. > > Back when we were thinking of supporting MSVC builds, we thought that > everything would need to be a C program, and so we rewrote the main > regression script in C, for example. However, now we have a build system > for MSVC we can see that its main infrastructure is in fact perl, so I > think what we need to do is to convert the MBCS and ECPG regression > drivers to perl rather than C - which should be far, far simpler task. > > I'm going to try to get these done and supported by the buildfarm by the > time we get to Beta. IIRC, Joachim had started working on the ecpg part. (Which is one of the reasons I haven't looked at it myself yet) Joachim, did you ever get anywhere with that? //Magnus
On Thu, May 17, 2007 at 10:14:45PM +0200, Magnus Hagander wrote: > IIRC, Joachim had started working on the ecpg part. (Which is one of the > reasons I haven't looked at it myself yet) > Joachim, did you ever get anywhere with that? I have ecpg tests somehow running under msvc but there are still some issues. Here are the problems that I encounter(ed): a) ecpg uses a program for each test that runs and generates output. Those programs need to be compiled: in MSVC we would either compile them on the fly while testing or generate a new project for every single executable. Since the second option is overkill, Magnus suggested to try compiling them on the fly with the command line compiler. This is ugly but seems to work. b) ecpg tests use several directories. This causes several problems, one of them being the resultmap format to be changed, it currently uses: float4/i.86-pc-mingw32=... we might want to change the / to some other character here. c) ecpg uses 3 different files per test. stderr, stdout and generated source. There is not only just one file to diff against an expected file. My approach was to have separate init and test routines for each "module" (backend regression tests, ecpg or whatever) and share the rest. The init routine sets up stuff and the test routine knows about the specific test logic. It generates lists of files to diff. The diffing and reporting is kept common of course. A problem here are alternative comparison files. We have the resultmap entries and we have "test_n.out". This has to be combined with ecpg having 3 files. We might want to use test-platform.stdout instead of test.stdout and test_1.stderr instead of test.stderr I have not yet sorted out this last item. Proposals welcome. I hope that I can fix it on the weekend or at least by the end of next week and then send my version to whoever is interested. Joachim
Joachim Wieland wrote: > On Thu, May 17, 2007 at 10:14:45PM +0200, Magnus Hagander wrote: >> IIRC, Joachim had started working on the ecpg part. (Which is one of the >> reasons I haven't looked at it myself yet) > >> Joachim, did you ever get anywhere with that? > > I have ecpg tests somehow running under msvc but there are still some issues. > Here are the problems that I encounter(ed): > > a) ecpg uses a program for each test that runs and generates output. Those > programs need to be compiled: in MSVC we would either compile them on the > fly while testing or generate a new project for every single executable. > Since the second option is overkill, Magnus suggested to try compiling them > on the fly with the command line compiler. This is ugly but seems to work. It might also be possible to generate a wildcard msbuild file for them, perhaps? (If you have Visual Studio 2005, you will have msbuild) Worth investigating, but this is probably the easiest thing to fix. > b) ecpg tests use several directories. This causes several problems, one of > them being the resultmap format to be changed, it currently uses: > > float4/i.86-pc-mingw32=... > > we might want to change the / to some other character here. Hmm. I guess doable if necessary. > c) ecpg uses 3 different files per test. stderr, stdout and generated > source. There is not only just one file to diff against an expected file. > > My approach was to have separate init and test routines for each "module" > (backend regression tests, ecpg or whatever) and share the rest. > The init routine sets up stuff and the test routine knows about the specific > test logic. It generates lists of files to diff. The diffing and reporting > is kept common of course. A problem here are alternative comparison files. > We have the resultmap entries and we have "test_n.out". This has to be > combined with ecpg having 3 files. We might want to use > > test-platform.stdout instead of test.stdout and > test_1.stderr instead of test.stderr > > I have not yet sorted out this last item. Proposals welcome. I hope that I > can fix it on the weekend or at least by the end of next week and then send > my version to whoever is interested. Hmm. Probably easier for someone who already knows how the ecpg tests actually work. I just keep breaking them ;-) However - do go ahead and send what you have to -patches for comments as a WIP patch, even if it's not ready. I'd certainly be interested at looking at what you have so far. //Magnus