Thread: Two weeks to feature freeze
We have less than two weeks to feature freeze. Win32 is still in an uncompleted state, and I haven't been able to return to it recently. Jan is working on getting exec() working, and hopefully someone can help me on signals. If I get those two done, I think I can tweek Win32 in minor ways during beta. I talked to Patrick about PITR, and with JR now back involved, he might get it done. Basically, we might get them both in, or it might be a disaster that we delayed beta for one month. I am heading to MIT now and will try to get all the outstanding patches in within the next few days. There are only a few left, mostly ones that appeared while I was applying the last patch backlog. I have also been asked to complete my O'Reilly slides by the end of next week, so I will have little time for Win32. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
> We have less than two weeks to feature freeze. Win32 is still in an > uncompleted state, and I haven't been able to return to it recently. > Jan is working on getting exec() working, and hopefully someone can help > me on signals. If I get those two done, I think I can tweek Win32 in > minor ways during beta. > > I talked to Patrick about PITR, and with JR now back involved, he might > get it done. > > Basically, we might get them both in, or it might be a disaster that we > delayed beta for one month. What about the nested transaction stuff? Do we have any "killer" features added to 7.4 that we can shout about? There's usually been one or two in the past...? Chris
On Thu, 2003-06-19 at 01:27, Christopher Kings-Lynne wrote: > > We have less than two weeks to feature freeze. Win32 is still in an > > uncompleted state, and I haven't been able to return to it recently. > > Jan is working on getting exec() working, and hopefully someone can help > > me on signals. If I get those two done, I think I can tweek Win32 in > > minor ways during beta. > > > > I talked to Patrick about PITR, and with JR now back involved, he might > > get it done. > > > > Basically, we might get them both in, or it might be a disaster that we > > delayed beta for one month. > > What about the nested transaction stuff? > > Do we have any "killer" features added to 7.4 that we can shout about? > There's usually been one or two in the past...? A quick glance at the TODO list shows a number of speed improvements in specific areas (IN, GROUP BY, Subselects in views), ARRAY improvements, some utility command improvements / additions, and a significant protocol update. The protocol update may not be flashy, but it is a large step forward in presenting a clean experience for developers using PostgreSQL (reduces chance of rare, unexpected, and difficult to find logic errors). If nothing else, it makes for an excellent cleanup release that rounds off some of the sharp corners (tab completion for schema elements in psql, schema dump in psql, fixed cluster support, transactional truncate, alter sequence, new regex code for fast MultiByte, etc). -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
On Wed, 2003-06-18 at 21:27, Christopher Kings-Lynne wrote: > Do we have any "killer" features added to 7.4 that we can shout about? > There's usually been one or two in the past...? Isn't the index growth problem solved in this release? I think that is a killer feature that solves a big problem for alot of people.
On Wed, Jun 18, 2003 at 09:59:17PM -0400, Matthew T. O'Connor wrote: > On Wed, 2003-06-18 at 21:27, Christopher Kings-Lynne wrote: > > Do we have any "killer" features added to 7.4 that we can shout about? > > There's usually been one or two in the past...? > > Isn't the index growth problem solved in this release? I think that is > a killer feature that solves a big problem for alot of people. Yes. That qualifies for "killer features" at least to some big users here. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Granting software the freedom to evolve guarantees only different results, not better ones." (Zygo Blaxell)
On Thu, Jun 19, 2003 at 09:27:22AM +0800, Christopher Kings-Lynne wrote: > > We have less than two weeks to feature freeze. > > What about the nested transaction stuff? I don't know if it will be completed before feature freeze... we can and will try, of course. Sadly, like most other people, I have lots of other things and can't give it the time I wish. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Porque francamente, si para saber manejarse a uno mismo hubiera que rendir examen... ¿Quién es el machito que tendría carnet?" (Mafalda)
"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> What about the nested transaction stuff?
With all due respect to Alvaro et al, I can't imagine that that will
make it into 7.4. (I have no confidence that PITR or Win32 native port
will make it either...)
> Do we have any "killer" features added to 7.4 that we can shout about?
We have a lot of pretty good stuff. You're not happy that the
performance of IN (subselect) has been fixed? That btree index bloat is
fixed (at least in large part, it remains to be seen whether the field
performance is all that we need...)?
In my opinion the project is not at a state where whizzy new Features
with a capital F are going to jump out of the woodwork. We are making
good advances in performance, reliability, SQL spec compliance, and
stuff like that, but fancy-sounding bullet points are hard to come by.
I can tell you that Red Hat's CCM group (the former Ars Digita) is
waiting with bated breath for 7.4, because it fixes a number of problems
(IN-subselect being one) that prevent 7.3 from being a serious
competitor to Oracle for their platform. 7.4 is a killer release for
them, and has been since about February, and they're getting tired of
waiting. I think a lot of other people are in the same situation,
even though they may not know it ;-)
We can't slip this puppy any more --- it's time to wrap her up and
push her out.
regards, tom lane
On Thu, 19 Jun 2003, Christopher Kings-Lynne wrote: > > We have less than two weeks to feature freeze. Win32 is still in an > > uncompleted state, and I haven't been able to return to it recently. > > Jan is working on getting exec() working, and hopefully someone can help > > me on signals. If I get those two done, I think I can tweek Win32 in > > minor ways during beta. > > > > I talked to Patrick about PITR, and with JR now back involved, he might > > get it done. > > > > Basically, we might get them both in, or it might be a disaster that we > > delayed beta for one month. > > What about the nested transaction stuff? > > Do we have any "killer" features added to 7.4 that we can shout about? > There's usually been one or two in the past...? I'm not sure if contrib/tsearch is a "killer" feature, but we hope to submit completely new version of tsearch V2 before July 1. Actually, we have stable code already used in some projects but currently lacking documentation. Several people are working on tutorial, reference guide. The problem is that Bruce seems is very overloaded and for sure he'll have many patches close to July 1. Is it possible to get rights to commit our changes ? > > Chris > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
On Thursday 19 June 2003 03:27, Christopher Kings-Lynne wrote: > Do we have any "killer" features added to 7.4 that we can shout about? We should not forget the availability of PostgreSQL companion products, like pgAdmin3 and PhpPgAdmin3. These two GUIs should be ready for release during July, although I am not in the shoes of the project leaders, Dave and Christopher and cannot speak of it officially. They are not part of the PostgreSQL distribution, but well could be. Providing a reliable bundle including PostgreSQL, a graphical GUI (pgAdmin3) and a web administration interface (PhpPgAdmin3) is a big news. This will convince entry users, who normally turn to MySQL, that PostgreSQL is the best choice. We are not downgrading features but upgrading user needs... When PostgreSQL win32 port is ready, all platforms and entry user needs will be covered. Isn't it a big news? Just my 2 cents... Best regards, Jean-Michel POURE
On Wed, 2003-06-18 at 23:07, Tom Lane wrote:
> "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> > What about the nested transaction stuff?
>
> With all due respect to Alvaro et al, I can't imagine that that will
> make it into 7.4. (I have no confidence that PITR or Win32 native port
> will make it either...)
>
Heres hoping for win32, that is a killer feature for so many people and
we're so close to it...
> > Do we have any "killer" features added to 7.4 that we can shout about?
>
> We have a lot of pretty good stuff. You're not happy that the
> performance of IN (subselect) has been fixed? That btree index bloat is
> fixed (at least in large part, it remains to be seen whether the field
> performance is all that we need...)?
>
I think the auto vacuum work will be pretty big, and I personally think
statement level triggers are pretty important too. (Which reminds me I
really need to start banging on those a bit more.)
> In my opinion the project is not at a state where whizzy new Features
> with a capital F are going to jump out of the woodwork. We are making
> good advances in performance, reliability, SQL spec compliance, and
> stuff like that, but fancy-sounding bullet points are hard to come by.
>
You mean like that other database that just recently added transaction
support ;-)
I do see a number of capital F features that haven't been done yet,
win32, replication, nested transactions... imho those features could
each warrant a development cycle on their own.
> I can tell you that Red Hat's CCM group (the former Ars Digita) is
> waiting with bated breath for 7.4, because it fixes a number of problems
> (IN-subselect being one) that prevent 7.3 from being a serious
> competitor to Oracle for their platform. 7.4 is a killer release for
> them, and has been since about February, and they're getting tired of
> waiting. I think a lot of other people are in the same situation,
> even though they may not know it ;-)
>
> We can't slip this puppy any more --- it's time to wrap her up and
> push her out.
>
Well, I suppose that history has shown that waiting on specific features
causes trouble with postgresql development, but I don't see why a
release can't be based around waiting for feature x as long as feature x
is being actively worked on by trusted developers who have an endgame in
sight.
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
Robert Treat <xzilla@users.sourceforge.net> writes:
> Well, I suppose that history has shown that waiting on specific features
> causes trouble with postgresql development, but I don't see why a
> release can't be based around waiting for feature x as long as feature x
> is being actively worked on by trusted developers who have an endgame in
> sight.
We have been led down that garden path before, and it's been a losing
proposition every time.
regards, tom lane
Maybe a better strategy would be to get a release out soon but not wait 6 months for another release which would contain the Win32 port and the PITR stuff (assuming those aren't done in time for this release). Just a thought. andrew Tom Lane wrote: > Robert Treat <xzilla@users.sourceforge.net> writes: >> Well, I suppose that history has shown that waiting on specific >> features causes trouble with postgresql development, but I don't see >> why a release can't be based around waiting for feature x as long as >> feature x is being actively worked on by trusted developers who have >> an endgame in sight. > > We have been led down that garden path before, and it's been a losing > proposition every time. >
On Thu, 2003-06-19 at 06:12, Andrew Dunstan wrote: > Maybe a better strategy would be to get a release out soon but not wait 6 > months for another release which would contain the Win32 port and the PITR > stuff (assuming those aren't done in time for this release). Thats what Justin was saying about this one, that it should be released early due to win32 being complete. I would expect, even once it compiles and runs (it doesn't yet) that it will need some good testing before being released in any form. Platform changes of this significance tend to introduce lots of new and unexpected issues. Tracking down several users for windows testing is a good idea. Rushing it out in an unknown state to get the testing isn't such a good idea in my mind. -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
Tom wrote: > > Do we have any "killer" features added to 7.4 that we can shout about? > > We have a lot of pretty good stuff. You're not happy that the > performance of IN (subselect) has been fixed? That btree index bloat is > fixed... For warehousing & reporting, "Add hash for evaluating GROUP BY aggregates" can be a killer feature. > 7.4 is a killer release for them, and has been since about February, > and they're getting tired of waiting. I think a lot of other people > are in the same situation, even though they may not know it ;-) A nightly reporting process that starts here at midnight each night takes about 12 hours on 7.3 and about 9 hours on 7.4alpha; possibly thanks to HashAggregates. While this may not sound like much, it means Marketing could see results when they arive at work, instead of waiting for afternoon. The perceived improvement of "ready before work" vs. "wait three hours" is a killer feature for this system. Ron
> We have a lot of pretty good stuff. You're not happy that the > performance of IN (subselect) has been fixed? All our code uses workaround now :) > That btree index bloat is > fixed (at least in large part, it remains to be seen whether the field > performance is all that we need...)? Yes, that's a good feature. > We can't slip this puppy any more --- it's time to wrap her up and > push her out. OK, let's do it - we've got full support for it in phpPgAdmin already :) Chris
On Thu, 19 Jun 2003, Robert Treat wrote: > Well, I suppose that history has shown that waiting on specific features > causes trouble with postgresql development, but I don't see why a > release can't be based around waiting for feature x as long as feature x > is being actively worked on by trusted developers who have an endgame in > sight. Everyone has an 'endgame in sight', at least when they ask for a release to be postponed ... but then their date keeps slipping, etc ... The thing is, if win32 is 'that close to being finished', then as soon as v7.4 is out, that code should be ready to throw in ... and the same for every other features that could 'postpone a release' ... I'd rather see the dev cycle shortened by a month, then extended ...
On Thu, 19 Jun 2003, Andrew Dunstan wrote:
>
> Maybe a better strategy would be to get a release out soon but not wait
> 6 months for another release which would contain the Win32 port and the
> PITR stuff (assuming those aren't done in time for this release).
>
> Just a thought.
And definitely in agreement here ... I'd rather see a shortened dev cycle
prompted by a big feature being added, then delaying a release because "oh
oh, I need another few weeks" that draws out when something unexpected
happens :(
> > andrew
>
> Tom Lane wrote:
> > Robert Treat <xzilla@users.sourceforge.net> writes:
> >> Well, I suppose that history has shown that waiting on specific
> >> features causes trouble with postgresql development, but I don't see
> >> why a release can't be based around waiting for feature x as long as
> >> feature x is being actively worked on by trusted developers who have
> >> an endgame in sight.
> >
> > We have been led down that garden path before, and it's been a losing
> > proposition every time.
> >
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postgresql.org
>
Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy
Systems Administrator @ hub.org
primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
On Thu, 19 Jun 2003, Oleg Bartunov wrote: > I'm not sure if contrib/tsearch is a "killer" feature, but we hope to > submit completely new version of tsearch V2 before July 1. Actually, we > have stable code already used in some projects but currently lacking > documentation. Several people are working on tutorial, reference guide. > The problem is that Bruce seems is very overloaded and for sure he'll > have many patches close to July 1. Is it possible to get rights to > commit our changes ? Is there a strong reason why tsearch isn't in gborg?
On Thu, 19 Jun 2003, The Hermit Hacker wrote: > On Thu, 19 Jun 2003, Oleg Bartunov wrote: > > > I'm not sure if contrib/tsearch is a "killer" feature, but we hope to > > submit completely new version of tsearch V2 before July 1. Actually, we > > have stable code already used in some projects but currently lacking > > documentation. Several people are working on tutorial, reference guide. > > The problem is that Bruce seems is very overloaded and for sure he'll > > have many patches close to July 1. Is it possible to get rights to > > commit our changes ? > > Is there a strong reason why tsearch isn't in gborg? > How gborg could help us submitting changes to pgsql CVS ? > ---------------------------(end of broadcast)--------------------------- > TIP 4: Don't 'kill -9' the postmaster > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
On Fri, 20 Jun 2003, Oleg Bartunov wrote: > > Is there a strong reason why tsearch isn't in gborg? > > > > How gborg could help us submitting changes to pgsql CVS ? It wouldn't ... is there a reason why tsearch needs to be in the pgsql CVS any more then, say, ODBC drivers, or the tcl interface, or the python interface, or ... ?
On Fri, 20 Jun 2003, The Hermit Hacker wrote: > On Fri, 20 Jun 2003, Oleg Bartunov wrote: > > > > Is there a strong reason why tsearch isn't in gborg? > > > > > > > How gborg could help us submitting changes to pgsql CVS ? > > It wouldn't ... is there a reason why tsearch needs to be in the pgsql CVS > any more then, say, ODBC drivers, or the tcl interface, or the python > interface, or ... ? because tsearch lives under pgsql source tree, in contrib directory though. I'd never asked for any rights (I have enough beside postgresq :), but practice of development life is so, that Bruce has no time to apply them in reasonable time. Last patch waited about 1 month. I'd like to see tsearch in contrib directory because we have direct benefit from that - rather often Tom change contrib sources to sync with his changes in main tree. This is the only real reason why I want to stay under postgres tree. I was afraid, and Bruce actually wrote he will be busy, we couldn't submit our new version. Freebsd has separate rights for core and ports. > > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
> On Fri, 20 Jun 2003, The Hermit Hacker wrote:
> Is there a strong reason why tsearch isn't in gborg?
I think text search is a pretty important facility that should
eventually be part of the core distribution. It's more likely to get
there from contrib than from gborg ...
regards, tom lane
On Fri, 20 Jun 2003, Tom Lane wrote: > > On Fri, 20 Jun 2003, The Hermit Hacker wrote: > > Is there a strong reason why tsearch isn't in gborg? > > I think text search is a pretty important facility that should > eventually be part of the core distribution. It's more likely to get > there from contrib than from gborg ... Why part of the core distribution, and not just left as a loadable module, like it is now?
> Why part of the core distribution, and not just left as a loadable module, > like it is now? The day I can go 'CREATE FULLTEXT INDEX ...' just like MySQL can I will be a very happy chappy... Chris
The Hermit Hacker kirjutas R, 20.06.2003 kell 08:28: > On Fri, 20 Jun 2003, Tom Lane wrote: > > > > On Fri, 20 Jun 2003, The Hermit Hacker wrote: > > > Is there a strong reason why tsearch isn't in gborg? > > > > I think text search is a pretty important facility that should > > eventually be part of the core distribution. It's more likely to get > > there from contrib than from gborg ... > > Why part of the core distribution, and not just left as a loadable module, > like it is now? I remember Tom saying that builtin functions calls are a lot faster than loadable C functions. If that can be fixed, then it *could* stay loadable. Also, having built-in full text indexing is very desirable. And I don't see any even nearly as good competing fulltext indexing modules anywhere. If we had to move something *out* of core in order to get tsearch in, then I personally would not mind if all geometry types go to gborg, but I'm sure there are some users who would mind ;) --------------- Hannu
On Thu, 19 Jun 2003, The Hermit Hacker wrote: > On Thu, 19 Jun 2003, Andrew Dunstan wrote: > > > > > Maybe a better strategy would be to get a release out soon but not wait > > 6 months for another release which would contain the Win32 port and the > > PITR stuff (assuming those aren't done in time for this release). > > > > Just a thought. > > And definitely in agreement here ... I'd rather see a shortened dev cycle > prompted by a big feature being added, then delaying a release because "oh > oh, I need another few weeks" that draws out when something unexpected > happens :( > >... I'm not sure why another delay is being considered. There's been a delay of a week because of the server problems hasn't there and wasn't the original delay only acceptable on the basis that that was that and there wasn't going to be another extension? -- Nigel Andrews
On Fri, 20 Jun 2003, Christopher Kings-Lynne wrote: > > Why part of the core distribution, and not just left as a loadable module, > > like it is now? > > The day I can go 'CREATE FULLTEXT INDEX ...' just like MySQL can I will be a > very happy chappy... > with new tserach v2 we're pretty close to that day. we need more testing, more suggestions and more documentation. There are several issues remains, mostly with core GiST not tsearch. The most important is concurrency support ! We've several times planned to work on it, but real life is rather complex thing :( > Chris > > > ---------------------------(end of broadcast)--------------------------- > TIP 8: explain analyze is your friend > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83
The Hermit Hacker wrote: > On Thu, 19 Jun 2003, Andrew Dunstan wrote: > > >>Maybe a better strategy would be to get a release out soon but not wait >>6 months for another release which would contain the Win32 port and the >>PITR stuff (assuming those aren't done in time for this release). >> >>Just a thought. > > > And definitely in agreement here ... I'd rather see a shortened dev cycle > prompted by a big feature being added, then delaying a release because "oh > oh, I need another few weeks" that draws out when something unexpected > happens :( Yep, this makes sense. Looks like it'll be PostgreSQL 7.4 being all the present improvements, but without PITR and Win32. Then, in a few months (hopefully less than 3) we'll have PostgreSQL 8.0, with both of those major features in it (and whatever other enhancements have been added). The only thing that makes me wince is that we have a protocol change at PostgreSQL 7.4 release instead of 8.0. It kind of doesn't sound right, having a protocol change in the "7 series", when we have an "8 series" coming up soon after. Oh well, so it's not perfect... ;-) Regards and best wishes, Justin Clift <snip>
On Fri, 2003-06-20 at 04:41, Nigel J. Andrews wrote:
>
> On Thu, 19 Jun 2003, The Hermit Hacker wrote:
>
> > On Thu, 19 Jun 2003, Andrew Dunstan wrote:
> >
> > >
> > > Maybe a better strategy would be to get a release out soon but not wait
> > > 6 months for another release which would contain the Win32 port and the
> > > PITR stuff (assuming those aren't done in time for this release).
> > >
> > > Just a thought.
> >
> > And definitely in agreement here ... I'd rather see a shortened dev cycle
> > prompted by a big feature being added, then delaying a release because "oh
> > oh, I need another few weeks" that draws out when something unexpected
> > happens :(
> >
> >...
>
> I'm not sure why another delay is being considered. There's been a delay of
> a week because of the server problems hasn't there and wasn't the original
> delay only acceptable on the basis that that was that and there wasn't going to
> be another extension?
>
There really isn't for this release. Any talk of delay is just a thought
on general policy for future releases.
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
On Fri, 2003-06-20 at 06:59, Justin Clift wrote:
> The Hermit Hacker wrote:
> > On Thu, 19 Jun 2003, Andrew Dunstan wrote:
> Yep, this makes sense. Looks like it'll be PostgreSQL 7.4 being all the
> present improvements, but without PITR and Win32. Then, in a few months
> (hopefully less than 3) we'll have PostgreSQL 8.0, with both of those
> major features in it (and whatever other enhancements have been added).
>
> The only thing that makes me wince is that we have a protocol change at
> PostgreSQL 7.4 release instead of 8.0. It kind of doesn't sound right,
> having a protocol change in the "7 series", when we have an "8 series"
> coming up soon after.
>
> Oh well, so it's not perfect...
>
...which is why I'd advocate making this release an 8.0 regardless of
win32 or pitr. I know it's old school to actually base versioning on
technical criteria instead of buzzwords, but there's no reason we have
to follow the corporate mold. Still, I'd rather not get into that debate
yet since I don't want to let the win32 guys off the hook yet!
win32 for the next release! go guys go!
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
Robert Treat <xzilla@users.sourceforge.net> writes:
> On Fri, 2003-06-20 at 06:59, Justin Clift wrote:
>> The only thing that makes me wince is that we have a protocol change at
>> PostgreSQL 7.4 release instead of 8.0.
> ...which is why I'd advocate making this release an 8.0 regardless of
> win32 or pitr.
<shrug> ... The backend will still talk to old clients, and libpq will
still talk to old backends, so I don't think the protocol change is
really going to cause a flag day for anyone. On a technical level it's
probably not an adequate reason to call this release 8.0.
On the other hand, I dislike the notion that we would call a release 8.0
simply because it now has a native Windows port. (And if there is a
short release cycle after this one, that might be about the only big new
thing there.) Considering that we aren't going to be recommending the
Windows port for production work, we should not let the release
numbering give the impression we think it's the greatest Postgres
feature ever to come down the pike.
I'm happy to keep calling 'em 7.* for the foreseeable future, myself.
regards, tom lane
And, actually, for some reason I hadn't thought of the tsearch as being another 'INDEX' type ... I crawl back over and be quiet now :) Oleg, as far as commits are concerned, I have no problems with extending the privileges to one of your guys for this, just email me seperately who, and I'll get it setup ... On Fri, 20 Jun 2003, Oleg Bartunov wrote: > On Fri, 20 Jun 2003, Christopher Kings-Lynne wrote: > > > > Why part of the core distribution, and not just left as a loadable module, > > > like it is now? > > > > The day I can go 'CREATE FULLTEXT INDEX ...' just like MySQL can I will be a > > very happy chappy... > > > > with new tserach v2 we're pretty close to that day. we need more testing, > more suggestions and more documentation. There are several issues remains, > mostly with core GiST not tsearch. The most important is concurrency support ! > We've several times planned to work on it, but real life is rather complex > thing :( > > > Chris > > > > > > ---------------------------(end of broadcast)--------------------------- > > TIP 8: explain analyze is your friend > > > > Regards, > Oleg > _____________________________________________________________ > Oleg Bartunov, sci.researcher, hostmaster of AstroNet, > Sternberg Astronomical Institute, Moscow University (Russia) > Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ > phone: +007(095)939-16-83, +007(095)939-23-83 > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
On Fri, 2003-06-20 at 10:42, Tom Lane wrote:
> Robert Treat <xzilla@users.sourceforge.net> writes:
> > On Fri, 2003-06-20 at 06:59, Justin Clift wrote:
> >> The only thing that makes me wince is that we have a protocol change at
> >> PostgreSQL 7.4 release instead of 8.0.
>
> > ...which is why I'd advocate making this release an 8.0 regardless of
> > win32 or pitr.
>
> <shrug> ... The backend will still talk to old clients, and libpq will
> still talk to old backends, so I don't think the protocol change is
> really going to cause a flag day for anyone. On a technical level it's
> probably not an adequate reason to call this release 8.0.
>
Can you give me an example of a technical change that would warrant a
major version bump?
> On the other hand, I dislike the notion that we would call a release 8.0
> simply because it now has a native Windows port. (And if there is a
> short release cycle after this one, that might be about the only big new
> thing there.) Considering that we aren't going to be recommending the
> Windows port for production work, we should not let the release
> numbering give the impression we think it's the greatest Postgres
> feature ever to come down the pike.
>
yep.
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
Robert, > Well, I suppose that history has shown that waiting on specific features > causes trouble with postgresql development, but I don't see why a > release can't be based around waiting for feature x as long as feature x > is being actively worked on by trusted developers who have an endgame in > sight. Ultimately, this is one of those "technical" vs. "marketing" questions ... whether to release now with a bunch of back-end features that the current users want, or to release later and include the features that we said were going to be in 7.4. And PostgreSQL is a technical project, not a marketing one. I know that, given MySQL's attempts to squeeze PostgreSQL out of the market, that there is a lot of desire to include at least one "killer feature" in each release to grab press coverage. But the PostgreSQL project can't let our technical decisions be governed by our PR, or we *become* MySQL. -- Josh Berkus Aglio Database Solutions San Francisco
Robert Treat <xzilla@users.sourceforge.net> writes:
> On Fri, 2003-06-20 at 10:42, Tom Lane wrote:
>> <shrug> ... The backend will still talk to old clients, and libpq will
>> still talk to old backends, so I don't think the protocol change is
>> really going to cause a flag day for anyone. On a technical level it's
>> probably not an adequate reason to call this release 8.0.
> Can you give me an example of a technical change that would warrant a
> major version bump?
Well, if we hadn't gotten the work done to make libpq still able to talk
to older backends, then we'd have had enough of a compatibility issue
that I think calling it 8.0 would have been a reasonable thing to do.
If you want a feature-with-a-capital-F reason for going to 8.0, there is
only one candidate Feature in my personal view, and that's a built-in
replication solution. That doesn't seem to be getting any nearer :-(
regards, tom lane
On Fri, 2003-06-20 at 11:21, Josh Berkus wrote:
> Robert,
>
> > Well, I suppose that history has shown that waiting on specific features
> > causes trouble with postgresql development, but I don't see why a
> > release can't be based around waiting for feature x as long as feature x
> > is being actively worked on by trusted developers who have an endgame in
> > sight.
>
> Ultimately, this is one of those "technical" vs. "marketing" questions ...
absolutely not. this is a x style of development vs. y style of
development discussion. many many projects, commercial and open source,
use a style of releasing based on features included in a given version.
In fact I'd be willing to say that the majority of open source projects
work this way, since open source projects generally aren't beholden to
release dates, giving developers the time they need to get specific
features done and release them when they are ready. as i prefaced in my
message, "history has shown us that waiting on specific features causes
trouble with postgresql development", but that doesn't mean we should
act as if this style of development doesn't exist.
> whether to release now with a bunch of back-end features that the current
> users want, or to release later and include the features that we said were
> going to be in 7.4. And PostgreSQL is a technical project, not a marketing
> one.
>
right, which is why core/hackers will decide both what gets into each
releases and when it's released. since i'm not outpacing tom or bruce
in my patch submissions i don't expect them to bend to my whims, but as
someone who follows and participates in postgresql development
regardless of an marketing aspects, i don't mind pointing out
alternatives.
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
Robert, > absolutely not. this is a x style of development vs. y style of > development discussion. many many projects, commercial and open source, > use a style of releasing based on features included in a given version. > In fact I'd be willing to say that the majority of open source projects > work this way, since open source projects generally aren't beholden to > release dates, giving developers the time they need to get specific > features done and release them when they are ready. as i prefaced in my > message, "history has shown us that waiting on specific features causes > trouble with postgresql development", but that doesn't mean we should > act as if this style of development doesn't exist. Ah. I see what you mean. Sorry for the misunderstanding. The thing is, from the perspective of *current* Postgres users, 7.4 has several "killer" features, some of which have been ready for 3 months. In fact, I just finished sending an e-mail to a client advising them to try 7.4 as soon as it is tested, becuase of hashaggs. So looked at from that perspective, our mistake was to try to cram too many features into 7.4 ... more than could possibly get done in 6-8 months. What's happening now is that the core group has decided, OK, we don't have 5-6 of the features we wanted to have, but we do have 10 other features, so maybe it's time to release. I'm not sure you're correct in the behaviour of the majoirty of OSS projects. Certainly if the Mozilla or OpenOffice.org projects are attaching specific release numbers to specific features, I haven't seen it. Linux does that, I guess, but that does result in a 2-year span between major releases -- and results in a lot of major features being included in "incremental" releases. But I think most OSS projects just release when they think they have enough tested features to justify a major release -- regardless of what those features are. Anybody here on other projects want to weigh in? Me, I'd be in favor of releasing more frequently ... I felt that we waited too long for 7.4. -- Josh Berkus Aglio Database Solutions San Francisco
Josh Berkus <josh@agliodbs.com> writes:
> So looked at from that perspective, our mistake was to try to cram too many
> features into 7.4 ... more than could possibly get done in 6-8 months.
It's more that we thought that all these features would get done in
about the same timeframe, and (not too surprisingly) some got done and
some didn't.
> Me, I'd be in favor of releasing more frequently ... I felt that we
> waited too long for 7.4.
Yeah, this is why I'm not in favor of slipping more.
Time was that we had a major release every 3 or 4 months. As the project
matures I think it's appropriate for the cycle to get slower: a lot of
low-hanging fruit is gone, so we have larger jobs to tackle, plus users
are using PG for larger databases and don't want to face major-version
changes too often. But I don't want it to get to be a year on average
between releases, at least not yet. 8 or 9 months seems reasonable, and
by that standard we're overdue.
regards, tom lane
The Hermit Hacker <scrappy@postgresql.org> writes: > On Thu, 19 Jun 2003, Robert Treat wrote: > >> Well, I suppose that history has shown that waiting on specific features >> causes trouble with postgresql development, but I don't see why a >> release can't be based around waiting for feature x as long as feature x >> is being actively worked on by trusted developers who have an endgame in >> sight. > > Everyone has an 'endgame in sight', at least when they ask for a > release to be postponed ... but then their date keeps slipping, etc > ... > > The thing is, if win32 is 'that close to being finished', then as > soon as v7.4 is out, that code should be ready to throw in ... and > the same for every other features that could 'postpone a release' > ... > > I'd rather see the dev cycle shortened by a month, then extended ... Why couldn't you just release the win32 version of 7.4 when it was finished. If it takes an extra month then that just gives you guys the chance to circulate *two* press releases. The Native Win32 port is likely to make a big enough splash all by itself. Jason
> -----Original Message----- > From: Jason Earl [mailto:jason.earl@simplot.com] > Sent: Friday, June 20, 2003 10:45 AM > To: The Hermit Hacker > Cc: Robert Treat; Tom Lane; Christopher Kings-Lynne; Bruce > Momjian; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > The Hermit Hacker <scrappy@postgresql.org> writes: > > > On Thu, 19 Jun 2003, Robert Treat wrote: > > > >> Well, I suppose that history has shown that waiting on specific > >> features causes trouble with postgresql development, but I > don't see > >> why a release can't be based around waiting for feature x > as long as > >> feature x is being actively worked on by trusted > developers who have > >> an endgame in sight. > > > > Everyone has an 'endgame in sight', at least when they ask for a > > release to be postponed ... but then their date keeps slipping, etc > > ... > > > > The thing is, if win32 is 'that close to being finished', > then as soon > > as v7.4 is out, that code should be ready to throw in ... > and the same > > for every other features that could 'postpone a release' ... > > > > I'd rather see the dev cycle shortened by a month, then extended ... > > Why couldn't you just release the win32 version of 7.4 when > it was finished. If it takes an extra month then that just > gives you guys the chance to circulate *two* press releases. > The Native Win32 port is likely to make a big enough splash > all by itself. A formal release needs a big testing effort. Two separate releases will double the work of validation.
Dann Corbit wrote: > > -----Original Message----- > > From: Jason Earl [mailto:jason.earl@simplot.com] > > Sent: Friday, June 20, 2003 10:45 AM > > To: The Hermit Hacker > > Cc: Robert Treat; Tom Lane; Christopher Kings-Lynne; Bruce > > Momjian; PostgreSQL-development > > Subject: Re: [HACKERS] Two weeks to feature freeze [...] > > Why couldn't you just release the win32 version of 7.4 when > > it was finished. If it takes an extra month then that just > > gives you guys the chance to circulate *two* press releases. > > The Native Win32 port is likely to make a big enough splash > > all by itself. > > A formal release needs a big testing effort. Two separate releases will > double the work of validation. That's true in the general case. But in this case we're talking about releasing for a completely new platform. That's much different than doing another release for the same platform set. The testing that needs to be done for the Win32 release has to be done separately *anyway*, so there's nothing lost by releasing the Win32 port separately. -- Kevin Brown kevin@sysexperts.com
"Dann Corbit" <DCorbit@connx.com> writes: >> >> Why couldn't you just release the win32 version of 7.4 when >> it was finished. If it takes an extra month then that just >> gives you guys the chance to circulate *two* press releases. >> The Native Win32 port is likely to make a big enough splash >> all by itself. > > A formal release needs a big testing effort. Two separate releases > will double the work of validation. There are several problems with that statement. The first is that PostgreSQL's "testing effort" happens right here on this mailing list. The various PostgreSQL hackers code stuff up, and we try and break it. There's very little /effort/ involved. People that want the new features go out on a limb and start using them. If they don't work, then they bring it up on the list. If they do work then very little gets said. As it now stands Tom Lane is on the record as stating that the new Win32 version isn't going to be ready for production anyhow. If anything the Win32 version *should* get released separately simply because we don't want people mistaking the Win32 version as being up to the PostgreSQL teams high standards. Those people that want the Win32 version to become production ready are going to have to risk their precious data. Otherwise, the Win32 version will likely remain a second class citizen forever. The fact of the matter is that the Win32 specific bits are the parts that are likely to break in the new port. If anything the Windows version will *benefit* from an earlier *nix release because the *nix users will chase down the bugs in the new PostgreSQL features. Once the *nix version is up to 7.4.2 (or so) then a Windows release of 7.4.2 should allow the PostgreSQL hackers to simply chase down Windows specific problems. Adding a new platform--especially a platform as diverse from the rest of PostgreSQL's supported platforms as Windows--is what adds the work. Testing the new platform is relatively easy. All you need to do is to start using the Win32 version with real live data. Jason
> -----Original Message----- > From: Jason Earl [mailto:jason.earl@simplot.com] > Sent: Friday, June 20, 2003 3:32 PM > To: Dann Corbit > Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <DCorbit@connx.com> writes: > >> > >> Why couldn't you just release the win32 version of 7.4 when > >> it was finished. If it takes an extra month then that just > >> gives you guys the chance to circulate *two* press releases. > >> The Native Win32 port is likely to make a big enough splash > >> all by itself. > > > > A formal release needs a big testing effort. Two separate releases > > will double the work of validation. > > There are several problems with that statement. The first is > that PostgreSQL's "testing effort" happens right here on this > mailing list. That's not exactly reassuring. There is no regression test that gets formal acceptance?! > The various PostgreSQL hackers code stuff up, > and we try and break it. There's very little /effort/ > involved. People that want the new features go out on a limb > and start using them. If they don't work, then they bring it > up on the list. If they do work then very little gets said. > > As it now stands Tom Lane is on the record as stating that > the new Win32 version isn't going to be ready for production > anyhow. If anything the Win32 version *should* get released > separately simply because we don't want people mistaking the > Win32 version as being up to the PostgreSQL teams high > standards. Those people that want the Win32 version to > become production ready are going to have to risk their > precious data. Otherwise, the Win32 version will likely > remain a second class citizen forever. > > The fact of the matter is that the Win32 specific bits are > the parts that are likely to break in the new port. If > anything the Windows version will *benefit* from an earlier > *nix release because the *nix users will chase down the bugs > in the new PostgreSQL features. Once the *nix version is up > to 7.4.2 (or so) then a Windows release of 7.4.2 should allow > the PostgreSQL hackers to simply chase down Windows specific problems. Then using the same logic, the new Windows version should wait indefinitely, since the *nix version will always be shaking out bugs. > Adding a new platform--especially a platform as diverse from > the rest of PostgreSQL's supported platforms as Windows--is > what adds the work. Testing the new platform is relatively > easy. All you need to do is to start using the Win32 version > with real live data. That is not testing. Using the world as your beta team seems to be a business model used by a few software giants that is largely frowned upon. I would think that there is an opportunity to do things differently. [Read 'properly']. We (at CONNX Solutions Inc.) have a formal release procedure that includes many tens of thousands of automated tests using dozens of different platforms. There are literally dozens of machines (I would guess 70 or so total) running around the clock for 7 days before we even know if we have a release candidate. The QA team is distinct from the development team, and if they say "FLOP!" the release goes nowhere. No formal release until QA passes it. If there is no procedure for PostgreSQL of this nature, then there really needs to be. I am sure that MySQL must have something in place like that. Their "Crash-Me" test suite shows (at least) that they have put a large effort into testing.
On Fri, Jun 20, 2003 at 03:39:47PM -0700, Dann Corbit wrote: > We (at CONNX Solutions Inc.) have a formal release procedure that > includes many tens of thousands of automated tests using dozens of > different platforms. [...] > > If there is no procedure for PostgreSQL of this nature, then there > really needs to be. I am sure that MySQL must have something in place > like that. Their "Crash-Me" test suite shows (at least) that they have > put a large effort into testing. The regression testing suite in Postgres is one of the things that impresses me about this software. It's very rare that a change is even committed to the main tree if a single regression test doesn't pass. When it does, a proper fix is quickly put in or the change reverted. It's even rare that patches with regression failures get posted in pgsql-patches. Regression tests are a very handy tool for contributors to check that their work is "safe". It's considered good practice to submit new tests when there's new functionality in a patch. There probably isn't such a gigantic effort like the one you describe, but there certainly _is_ a testing procedure. There's probably room for improvement, of course, but we don't want the tests to take a full week to complete, IMHO. It would be nice to have a system which could receive a patch and compile and verify that it passes the tests before it goes to Bruce's queue; or compile on multiple platforms to check for portability problems, for example. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Uno puede defenderse de los ataques; contra los elogios se esta indefenso"
"Dann Corbit" <DCorbit@connx.com> writes: >> -----Original Message----- >> From: Jason Earl [mailto:jason.earl@simplot.com] >> Sent: Friday, June 20, 2003 3:32 PM >> To: Dann Corbit >> Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development >> Subject: Re: [HACKERS] Two weeks to feature freeze >> >> >> "Dann Corbit" <DCorbit@connx.com> writes: >> >> >> >> Why couldn't you just release the win32 version of 7.4 when >> >> it was finished. If it takes an extra month then that just >> >> gives you guys the chance to circulate *two* press releases. >> >> The Native Win32 port is likely to make a big enough splash >> >> all by itself. >> > >> > A formal release needs a big testing effort. Two separate releases >> > will double the work of validation. >> >> There are several problems with that statement. The first is >> that PostgreSQL's "testing effort" happens right here on this >> mailing list. > > That's not exactly reassuring. There is no regression test that > gets formal acceptance?! Yes, there are regression tests, and new tests get invented all of the time whenever the real world finds new bugs. Regression tests are excellent for making sure that you don't make the same mistake twice, but they aren't a substitute for handing the code over to actual end users. >> The various PostgreSQL hackers code stuff up, >> and we try and break it. There's very little /effort/ >> involved. People that want the new features go out on a limb >> and start using them. If they don't work, then they bring it >> up on the list. If they do work then very little gets said. >> >> As it now stands Tom Lane is on the record as stating that >> the new Win32 version isn't going to be ready for production >> anyhow. If anything the Win32 version *should* get released >> separately simply because we don't want people mistaking the >> Win32 version as being up to the PostgreSQL teams high >> standards. Those people that want the Win32 version to >> become production ready are going to have to risk their >> precious data. Otherwise, the Win32 version will likely >> remain a second class citizen forever. >> >> The fact of the matter is that the Win32 specific bits are >> the parts that are likely to break in the new port. If >> anything the Windows version will *benefit* from an earlier >> *nix release because the *nix users will chase down the bugs >> in the new PostgreSQL features. Once the *nix version is up >> to 7.4.2 (or so) then a Windows release of 7.4.2 should allow >> the PostgreSQL hackers to simply chase down Windows specific problems. > > Then using the same logic, the new Windows version should wait > indefinitely, since the *nix version will always be shaking out > bugs. That's not true at all. Despite the excellent work by the PostgreSQL team, and despite the beta testing that will be done by volunteers, if history repeats itself, there will be problems with version 7.4.0, even on platforms that have been well supported by PostgreSQL forever. I am not saying that we should hold off indefinitely on the Win32 port, I am simply saying that it probably wouldn't hurt to shake out the normal .0 release bugs before throwing the unique Win32 bugs into the mix. My guess is that reported Win32 bugs are going blamed on the Win32 specific bits at first no matter what happens. Unless the bug can be demonstrated on a *nix version it will be assumed that the problem is a shortcoming of the Win32 specific code. That's just common sense. >> Adding a new platform--especially a platform as diverse from >> the rest of PostgreSQL's supported platforms as Windows--is >> what adds the work. Testing the new platform is relatively >> easy. All you need to do is to start using the Win32 version >> with real live data. > > That is not testing. Using the world as your beta team seems to be > a business model used by a few software giants that is largely > frowned upon. I would think that there is an opportunity to do > things differently. [Read 'properly']. Hmm... I must have missed the huge corporation paying for in house testing of PostgreSQL. In the Free Software world the "beta team" is all of those people that need the new features so badly that they are willing to risk their own data and hardware testing it. You might not like the way that this sounds, but in practice it works astoundingly well. Chances are you can't name 25 pieces of commercial software that run on the wide array of hardware platforms and OSes as PostgreSQL, and PostgreSQL has a earned a well deserved reputation for being a solid piece of software. Clearly the PostgreSQL team is doing *something* right. > We (at CONNX Solutions Inc.) have a formal release procedure that > includes many tens of thousands of automated tests using dozens of > different platforms. There are literally dozens of machines (I > would guess 70 or so total) running around the clock for 7 days > before we even know if we have a release candidate. The QA team is > distinct from the development team, and if they say "FLOP!" the > release goes nowhere. No formal release until QA passes it. And yet when you release the software your customers invariably find bugs, don't they? Don't get me wrong. I am all for testing, regression tests, and such, but the fact of the matter is that there simply is no way that a centralized authority could afford to really test PostgreSQL on even a fraction of the supported platforms and configurations. The way it stands now the PostgreSQL teams gets the best testbed you could hope for (the world) for the price of hosting a few web and FTP servers (thanks Marc). PostgreSQL betas almost certainly gest tested on an order of magnitude more systems than the 70 that you boast about. PostgreSQL gets tested on everything from Sony Playstations to AMD Opterons to IBM mainframes. Heck, there are probably more than 70 machines running CVS versions of PostgreSQL right this minute (Marc, any download numbers to back this up?). More importantly, PostgreSQL gets tested on a wide variety of real world tasks, and not some lab application or some test scripts. Like I have mentioned several times before. PostgreSQL gets tested by folks that put their actual data into the beta versions and try it out. Even with this sort of testing, however, bugs still make it into the release version. Even with a large group of beta testers we simply can't test all of the possible ways that the software might get used on every available platform. > If there is no procedure for PostgreSQL of this nature, then there > really needs to be. I am sure that MySQL must have something in place > like that. Their "Crash-Me" test suite shows (at least) that they have > put a large effort into testing. Yow! Have you read the crash-me script. It's possible that they have improved dramatically in the year or so since I last took a look at them, but it used to be that MySQL's crash-me scripts were the worst amalgamation of marketeering and poor relational theory ever conceived by the human mind. Basically the crash-me scripts were nothing more than an attempt to put MySQL's competition in the worst light possible. Basically any time a competitor differed from MySQL an error would be generated (despite the fact that it was very likely that it was MySQL that was wrong). MySQL even tried to pawn this single-process monstrosity off as a "benchmark." What a laugh. It was a perfectly valid benchmark if your database was designed to be used by one user at a time and one of your biggest criteria was the time it took to create a valid connection from a perl script. PostgreSQL's regression tests (IMHO) are much better than MySQL's crash-me scripts. Jason
> -----Original Message----- > From: Jason Earl [mailto:jason.earl@simplot.com] > Sent: Friday, June 20, 2003 4:43 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <DCorbit@connx.com> writes: > > >> -----Original Message----- > >> From: Jason Earl [mailto:jason.earl@simplot.com] > >> Sent: Friday, June 20, 2003 3:32 PM > >> To: Dann Corbit > >> Cc: Jason Earl; The Hermit Hacker; PostgreSQL-development > >> Subject: Re: [HACKERS] Two weeks to feature freeze > >> > >> > >> "Dann Corbit" <DCorbit@connx.com> writes: > >> >> > >> >> Why couldn't you just release the win32 version of 7.4 > when it was > >> >> finished. If it takes an extra month then that just gives you > >> >> guys the chance to circulate *two* press releases. > >> >> The Native Win32 port is likely to make a big enough splash > >> >> all by itself. > >> > > >> > A formal release needs a big testing effort. Two > separate releases > >> > will double the work of validation. > >> > >> There are several problems with that statement. The first is > >> that PostgreSQL's "testing effort" happens right here on this > >> mailing list. > > > > That's not exactly reassuring. There is no regression test > that gets > > formal acceptance?! > > Yes, there are regression tests, and new tests get invented > all of the time whenever the real world finds new bugs. > Regression tests are excellent for making sure that you don't > make the same mistake twice, but they aren't a substitute for > handing the code over to actual end users. After testing & QA, there is a beta period. You don't hand beta code over to actual end users. In the corporate world it would be a clear case of both negligence and incompetence. > >> The various PostgreSQL hackers code stuff up, > >> and we try and break it. There's very little /effort/ > >> involved. People that want the new features go out on a limb > >> and start using them. If they don't work, then they bring it > >> up on the list. If they do work then very little gets said. > >> > >> As it now stands Tom Lane is on the record as stating that > >> the new Win32 version isn't going to be ready for production > >> anyhow. If anything the Win32 version *should* get released > >> separately simply because we don't want people mistaking the > >> Win32 version as being up to the PostgreSQL teams high > >> standards. Those people that want the Win32 version to > >> become production ready are going to have to risk their > >> precious data. Otherwise, the Win32 version will likely > >> remain a second class citizen forever. > >> > >> The fact of the matter is that the Win32 specific bits are > >> the parts that are likely to break in the new port. If > >> anything the Windows version will *benefit* from an earlier > >> *nix release because the *nix users will chase down the bugs > >> in the new PostgreSQL features. Once the *nix version is up > >> to 7.4.2 (or so) then a Windows release of 7.4.2 should allow > >> the PostgreSQL hackers to simply chase down Windows > specific problems. > > > > Then using the same logic, the new Windows version should wait > > indefinitely, since the *nix version will always be shaking > out bugs. > > That's not true at all. Despite the excellent work by the > PostgreSQL team, and despite the beta testing that will be > done by volunteers, if history repeats itself, there will be > problems with version 7.4.0, even on platforms that have been > well supported by PostgreSQL forever. I am not saying that we > should hold off indefinitely on the Win32 port, I am simply > saying that it probably wouldn't hurt to shake out the normal > .0 release bugs before throwing the unique Win32 bugs into the mix. > > My guess is that reported Win32 bugs are going blamed on the > Win32 specific bits at first no matter what happens. Unless > the bug can be demonstrated on a *nix version it will be > assumed that the problem is a shortcoming of the Win32 > specific code. That's just common sense. You are right that a new feature will add new bugs. I am saying that the Win32 port is a new feature that will need a shakedown, but the shakedown should occur in the testing and beta phase, like any other feature. > >> Adding a new platform--especially a platform as diverse from > >> the rest of PostgreSQL's supported platforms as Windows--is > >> what adds the work. Testing the new platform is relatively > >> easy. All you need to do is to start using the Win32 version > >> with real live data. > > > > That is not testing. Using the world as your beta team > seems to be a > > business model used by a few software giants that is > largely frowned > > upon. I would think that there is an opportunity to do things > > differently. [Read 'properly']. > > Hmm... I must have missed the huge corporation paying for in > house testing of PostgreSQL. In the Free Software world the > "beta team" is all of those people that need the new features > so badly that they are willing to risk their own data and > hardware testing it. I don't see how this model can possibly succeed then. You can't just hope that your end users will: 1. Exhaustively test 2. Accurately report the findings > You might not like the way that this > sounds, but in practice it works astoundingly well. Chances > are you can't name 25 pieces of commercial software that run > on the wide array of hardware platforms and OSes as > PostgreSQL, and PostgreSQL has a earned a well deserved > reputation for being a solid piece of software. Clearly the > PostgreSQL team is doing > *something* right. I don't argue that PostgreSQL is a good piece of software. I happen to like it very much and have been a staunch advocate for its use with our commercial products as well as in house. What I am saying is that it may be possible to improve the process. If the corporate world knew that the only testing applied to PostgreSQL was ad-hoc, I doubt that it would be accepted at all anywhere. The fact that PostgreSQL does succeed shows that the installed base of users must be highly intelligent and highly motivated. > > We (at CONNX Solutions Inc.) have a formal release procedure that > > includes many tens of thousands of automated tests using dozens of > > different platforms. There are literally dozens of > machines (I would > > guess 70 or so total) running around the clock for 7 days before we > > even know if we have a release candidate. The QA team is distinct > > from the development team, and if they say "FLOP!" the release goes > > nowhere. No formal release until QA passes it. > > And yet when you release the software your customers > invariably find bugs, don't they? Our beta customers do help us to find bugs. Bugs reported by customers for released products are extremely rare. The total issue count is 2495 in our bug tracking database (active since the late 1980's). There are 82 issues found by the customers in that list, and 7 with an issue ID over 2000 (recent issues). Our code base is several hundred thousand lines of code, and we have many thousands of customers world-wide. When I first started here, testing was less rigorous, and largely done by the programmers instead of separate testing teams. Since formal testing procedures have been established, technical support incidents have gone way down and quality has gone way up. > Don't get me wrong. I am all for testing, regression tests, > and such, but the fact of the matter is that there simply is > no way that a centralized authority could afford to really > test PostgreSQL on even a fraction of the supported platforms > and configurations. The way it stands now the PostgreSQL > teams gets the best testbed you could hope for (the world) > for the price of hosting a few web and FTP servers (thanks Marc). > > PostgreSQL betas almost certainly gest tested on an order of > magnitude more systems than the 70 that you boast about. Maybe it does. Maybe it doesn't. You have no way of knowing, since no formal reporting procedure exists. > PostgreSQL gets tested on everything from Sony Playstations > to AMD Opterons to IBM mainframes. Heck, there are probably > more than 70 machines running CVS versions of PostgreSQL > right this minute (Marc, any download numbers to back this > up?). If your count all the end-users workstations, our products have millions of seats. We run on UNIX (Solaris, Linux, AIX, Tru64, HP/UX, etc.) and OpenVMS and MVS and Win32 and OS/400 and others. As you can well imagine, we *must* have an enormous testing effort. > More importantly, PostgreSQL gets tested on a wide > variety of real world tasks, and not some lab application or > some test scripts. Spoken like a programmer. Yes, real world data *always* turns up things that neither the testers nor the programmers imagined. But a huge and comprehensive conformance testing effort will turn up 99% of the problems. > Like I have mentioned several times > before. PostgreSQL gets tested by folks that put their actual > data into the beta versions and try it out. Even with this > sort of testing, however, bugs still make it into the release > version. Bug cost as a function of discovery stage is exponential. 1. Discovered in design phase: nearly free to fix (designer sees bug, designer fixes bug) 2. Discovered in unit test phase: very cheap to fix (programmer sees bug, programmer fixes bug) 3. Discovered in integration test phase: inexpensive to fix (other engineers become involved) 4. Discovered in beta test phase: expensive to fix (customers, tech-support, sales, programmers, engineers involved) 5. Discovered in release: catastrophic cost to fix (as above, but now every single customer must be upgraded, tens of thousands of dollars lost, minimum -- possibly millions) > Even with a large group of beta testers we simply > can't test all of the possible ways that the software might > get used on every available platform. 100% code coverage is impossible. Program proving is impossible. 0% defect code delivery is impossible. But you should try to approach the ideal as closely as can be attained. > > If there is no procedure for PostgreSQL of this nature, then there > > really needs to be. I am sure that MySQL must have > something in place > > like that. Their "Crash-Me" test suite shows (at least) that they > > have put a large effort into testing. > > Yow! Have you read the crash-me script. It's possible that > they have improved dramatically in the year or so since I > last took a look at them, but it used to be that MySQL's > crash-me scripts were the worst amalgamation of marketeering > and poor relational theory ever conceived by the human mind. The tests are good tests. They cover a wide range of features and functions and discover if you can cause permanent damage to a database by simply performing end-user queries. The scripts are a bit hokey, but it isn't all that difficult to get them to run. > Basically the crash-me scripts were nothing more than an > attempt to put MySQL's competition in the worst light > possible. I disagree. In fact, in their matrix, PostgreSQL looks remarkably good. In fact, I would choose it over MySQL, if the only examination made was of the information contained in the matrix (but nobody would really drive a decision based on that data alone). > Basically any time a competitor differed from > MySQL an error would be generated (despite the fact that it > was very likely that it was MySQL that was wrong). This is unfair and untrue. (I have no connection whatsoever with the MySQL group, BTW). > MySQL even tried to pawn this single-process monstrosity off > as a "benchmark." What a laugh. It was a perfectly valid > benchmark if your database was designed to be used by one > user at a time and one of your biggest criteria was the time > it took to create a valid connection from a perl script. You can call it a conformance benchmark. It is not touted as a performance benchmark. And nobody would fall for it if it were, since it does not contain time information. > PostgreSQL's regression tests (IMHO) are much better than > MySQL's crash-me scripts. They are less thorough in coverage, but not too bad. Here is what I suggest: PostgreSQL has an excellent programming team. Why not try to recruit a similar testing team? I think it would strongly differentiate the tool set from similar free stuff. Perhaps all that is needed is some sort of automated, formal reporting procedure. For example, a large test set might be created that runs a thorough regression feature list. When the test completes, a data file is emailed to some central repository, parsed, and stored in a database. If the end-users can simply start some shell script and take off for the weekend, then it would be possible to collect a large volume of data.
Jason Earl <jason.earl@simplot.com> writes:
> The fact of the matter is that the Win32 specific bits are the parts
> that are likely to break in the new port.
Actually, what scares me about this is the probability that the Win32
port will break other platforms. The changes look to be invasive enough
to create a nontrivial risk of that.
For comparison, look at the CVS history of the recent efforts to support
IPv6 connections. Those patches have broken both IPv4 and Unix-socket
connections at different times, and are still a source of ongoing build
problems on some platforms, plus who-knows-what problems yet to be found
on platforms that aren't used by the bleeding-edge-CVS folk. I predict
that the tweaks needed to support Windows' lack of a fork() primitive
will be far worse.
BTW, I would not approve of a response along the lines of "can't you
#ifdef to the point that there are no code changes in the Unix builds?"
No you can't, unless you want to end up with an unmaintainable mess
of #ifdef spaghetti. The thing that makes this hard is the tradeoff
between making the code readable and maintainable (which requires
sharing as much code as possible across platforms) vs isolating
platform-specific considerations. Programming at this level is not
a science but an art form, and it's very hard to get it right the first
time --- especially when none of us have access to all the platforms
that the code must ultimately work on.
regards, tom lane
"Dann Corbit" <DCorbit@connx.com> writes:
> If there is no procedure for PostgreSQL of this nature, then there
> really needs to be.
Are you volunteering to create it? Step right up.
> I am sure that MySQL must have something in place
> like that. Their "Crash-Me" test suite shows (at least) that they have
> put a large effort into testing.
...ROTFL... Crash-Me is not a regression test. It is a marketing
effort.
regards, tom lane
Alvaro Herrera <alvherre@dcc.uchile.cl> writes:
> It would be nice to have a system which could receive a patch and
> compile and verify that it passes the tests before it goes to Bruce's
> queue; or compile on multiple platforms to check for portability
> problems, for example.
It happens not infrequently that Bruce commits some patch that fails the
regression tests. Whereupon he gets chewed out by whoever notices it
first :-). I've been guilty of the same on occasion, even though I try
hard to avoid it. If the regression tests took two seconds to run I'm
sure we'd both set up scripts to ensure we *never* commit without
regression testing first. On the other hand, if they took a week to run
you can be damn sure that most commits would go in without any
regression testing. I think that our existing tests are a fairly happy
medium --- they catch most stuff, and the stuff they don't catch is in
my opinion stuff that an automated test is unlikely to catch. (I do add
things to the regression tests whenever something shows up that looks
like it should have been caught.)
Another point is that passing on one platform doesn't ensure passing on
another. Here we really rely on the willingness of the pghackers
community to update to CVS tip regularly and run the regression tests
when they do. Again, tests that take a couple minutes to run are ideal;
if they took a week then the uptake would drop to zero, and we'd not be
ahead.
regards, tom lane
Jason Earl <jason.earl@simplot.com> writes:
> Hmm... I must have missed the huge corporation paying for in house
> testing of PostgreSQL. In the Free Software world the "beta team" is
> all of those people that need the new features so badly that they are
> willing to risk their own data and hardware testing it.
I don't have a lot of faith in huge automated test efforts. They're
great at ensuring you don't make the same mistakes you made once before,
but in my experience the nastiest bugs are the ones you haven't seen
before and would never in a million years have dreamed to test for.
Thus, the best test team is a bunch of people doing unplanned things
with the software, on a wide variety of platforms...
regards, tom lane
Tom Lane wrote: >Alvaro Herrera <alvherre@dcc.uchile.cl> writes: > > >>It would be nice to have a system which could receive a patch and >>compile and verify that it passes the tests before it goes to Bruce's >>queue; or compile on multiple platforms to check for portability >>problems, for example. >> *snip* >Another point is that passing on one platform doesn't ensure passing on >another. Here we really rely on the willingness of the pghackers >community to update to CVS tip regularly and run the regression tests >when they do. Again, tests that take a couple minutes to run are ideal; >if they took a week then the uptake would drop to zero, and we'd not be >ahead. > > Have you considered something similar to the Mozilla tinderbox approach where you have a daemon checkout the cvs, compile, run regression tests, and report a status or be able to report a status?
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Friday, June 20, 2003 8:36 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <DCorbit@connx.com> writes: > > If there is no procedure for PostgreSQL of this nature, then there > > really needs to be. > > Are you volunteering to create it? Step right up. No. And as an outsider, I rather doubt if any procedures I developed would be taken very seriously. If such procedures are to be developed, I suspect that they will have to be developed from within if they are to be successful. This would be a good start: A. Combine:1. Your regression test2. Crashme (or some rough equivalent if you don't like it)3. The NIST validation testsuite B. Automate:1. Installation of the tests2. Execution of the tests3. Transfer of the test results to a repository4. Analysisof the test results C. Assign:1. Criteria for acceptance of a build for release2. Authority for acceptance of a build for release3. Delegationrules for issue resolution4. Procedures for issue resolution > > I am sure that MySQL must have something in place > > like that. Their "Crash-Me" test suite shows (at least) that they > > have put a large effort into testing. > > ...ROTFL... Crash-Me is not a regression test. It is a > marketing effort. Let's see... Their marketing effort checks for STANDARDS conformance against over several hundred distinct, important properties. Their marketing effort checks for a number of interesting and valuable extensions. Their marketing effort checks for system safety in a manner that is better than anything I have ever seen from a commercial vendor. And the PostgreSQL regression test is superior in what ways? Look at this: http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on Their marketing effort makes PostgreSQL look superior to MySQL in most areas. If it is a marketing effort, then we must applaud them for their honesty.
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Friday, June 20, 2003 8:58 PM > To: Jason Earl > Cc: Dann Corbit; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > Jason Earl <jason.earl@simplot.com> writes: > > Hmm... I must have missed the huge corporation paying for in house > > testing of PostgreSQL. In the Free Software world the > "beta team" is > > all of those people that need the new features so badly > that they are > > willing to risk their own data and hardware testing it. > > I don't have a lot of faith in huge automated test efforts. > They're great at ensuring you don't make the same mistakes > you made once before, but in my experience the nastiest bugs > are the ones you haven't seen before and would never in a > million years have dreamed to test for. This is true if and only if the test design is poor. > Thus, the best test > team is a bunch of people doing unplanned things with the > software, on a wide variety of platforms... That is the worst possible test plan. It totally lacks organization and there is no hint to define when the feature set has been covered. Ad hoc testing is a useful addition, but it cannot replace all the standard tests that have been used by the industry for decades. If you run literally hundreds of tests designed to ensure that your product conforms to ANSI/ISO standards then the bugs that are missed will be few and far between. Unless you are bad at designing tests. Designing tests is busywork. Desiging tests is boring. Nobody wants to design tests, let alone interpret the results and define correct baselines. But testing is very, very important.
--- Dann Corbit <DCorbit@connx.com> wrote: > Why couldn't you just release the win32 version of 7.4 > when it was finished. I agree. Don't delay *nix release because of win32 port is not ready. To many users win32 port is of marginal importance anyway. __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
Thomas Swan <tswan@idigx.com> writes:
> Have you considered something similar to the Mozilla tinderbox approach
> where you have a daemon checkout the cvs, compile, run regression tests,
> and report a status or be able to report a status?
Tinderbox is pretty cool. Who wants to set it up?
regards, tom lane
"Dann Corbit" <DCorbit@connx.com> writes:
>> ...ROTFL... Crash-Me is not a regression test. It is a
>> marketing effort.
> Their marketing effort checks for STANDARDS conformance against over
> several hundred distinct, important properties.
If you'd not spelled STANDARDS in caps I'd not have taken the trouble to
respond ... but I suggest you stop to count exactly how many of their
bullet points are actually grounded in the SQL standard, how many are
not, and how many are in fact counter to spec but agree with MySQL's
deviations from spec (of course those show as green for MySQL's version
of reality, and as "failures" for spec-compliant databases).
I have been through crash-me in some detail, and it left a very bad
taste in my mouth. Don't bother holding it up as an example of good
practice.
regards, tom lane
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Friday, June 20, 2003 9:19 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <DCorbit@connx.com> writes: > >> ...ROTFL... Crash-Me is not a regression test. It is a > >> marketing effort. > > > Their marketing effort checks for STANDARDS conformance > against over > > several hundred distinct, important properties. > > If you'd not spelled STANDARDS in caps I'd not have taken the > trouble to respond Sorry for shouting. >... but I suggest you stop to count > exactly how many of their bullet points are actually grounded > in the SQL standard, how many are not, and how many are in > fact counter to spec but agree with MySQL's deviations from > spec (of course those show as green for MySQL's version of > reality, and as "failures" for spec-compliant databases). > > I have been through crash-me in some detail, and it left a > very bad taste in my mouth. Don't bother holding it up as an > example of good practice. Every single test in their list is interesting and useful. I see several hundred things which I recognize as part of the ANSI/ISO SQL Standard. I have set up and run the tests. I did not go into great detail (as you have done) to ensure that they were really testing what they claimed to test and that correct interpretation of the results was made in each case. If they have done something underhanded, then they should be called out onto the carpet for it. In any case, the general outline of what they are doing is a very good idea. If it can be improved upon, then that would be an excellent idea.
On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote: Citing Tom Lane: > > I have been through crash-me in some detail, and it left a > > very bad taste in my mouth. Don't bother holding it up as an > > example of good practice. > > Every single test in their list is interesting and useful. At least on the version I just saw there are several results with Postgres that are weird (table names > 500 chars?). Other things tested are clearly wrong (things that are = NULL, dates like '00-00-0000'); results for Postgres that are wrong probably because they are trying a weird syntax. Etc. Things like that drive the credibility of the whole thing to the floor. Maybe something like this should exist for Postgres, but it's not crash-me. Maybe the NIST compliance test is adequate. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "La conclusion que podemos sacar de esos estudios es que no podemos sacar ninguna conclusion de ellos" (Tanenbaum)
> -----Original Message----- > From: Alvaro Herrera [mailto:alvherre@dcc.uchile.cl] > Sent: Friday, June 20, 2003 10:00 PM > To: Dann Corbit > Cc: Tom Lane; Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote: > > Citing Tom Lane: > > > I have been through crash-me in some detail, and it left a > > > very bad taste in my mouth. Don't bother holding it up as an > > > example of good practice. > > > > Every single test in their list is interesting and useful. > > At least on the version I just saw there are several results > with Postgres that are weird (table names > 500 chars?). It does get silly at a point, but I have seen systems with 128 characters for table names, column names, etc. Some people seem to like it. Not me. Too much typing. > Other things tested are clearly wrong (things that are = > NULL, Sounds like testing for the existence of a bug. X = NULL X <= NULL X >= NULL Etc. must always test false, regardless of the contents of X. Test for equality with NULL is a conformance error if NULL == NULL returns true. > dates like '00-00-0000'); Not sure what that might even mean. > results for Postgres that are > wrong probably because they are trying a weird syntax. Etc. > > Things like that drive the credibility of the whole thing to > the floor. Maybe something like this should exist for > Postgres, but it's not crash-me. Maybe the NIST compliance > test is adequate. So far, I have seen three problems pointed out (out of 600+ tests). That's 0.5% defects. Why not just drop the stupid tests, or bend them to test for what they ought to be testing. Besides those three, what other tests are bogus and why?
> We (at CONNX Solutions Inc.) have a formal release procedure that > includes many tens of thousands of automated tests using dozens of > different platforms. There are literally dozens of machines (I would > guess 70 or so total) running around the clock for 7 days before we even > know if we have a release candidate. The QA team is distinct from the > development team, and if they say "FLOP!" the release goes nowhere. No > formal release until QA passes it. PostgreSQL has a comprehensive regression suite that is run by the developers all the time... > If there is no procedure for PostgreSQL of this nature, then there > really needs to be. I am sure that MySQL must have something in place > like that. Their "Crash-Me" test suite shows (at least) that they have > put a large effort into testing. No, it means they've put a crap effort into trying to make other databases look bad... Chris
> -----Original Message----- > From: Christopher Kings-Lynne [mailto:chriskl@familyhealth.com.au] > Sent: Friday, June 20, 2003 10:14 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > > We (at CONNX Solutions Inc.) have a formal release procedure that > > includes many tens of thousands of automated tests using dozens of > > different platforms. There are literally dozens of > machines (I would > > guess 70 or so total) running around the clock for 7 days before we > > even know if we have a release candidate. The QA team is distinct > > from the development team, and if they say "FLOP!" the release goes > > nowhere. No formal release until QA passes it. > > PostgreSQL has a comprehensive regression suite that is run > by the developers all the time... If you mean the one that comes with PostgreSQL, then I think the MySQL test is better. The PostgreSQL test seems to focus more on extensions than anything else. > > If there is no procedure for PostgreSQL of this nature, then there > > really needs to be. I am sure that MySQL must have > something in place > > like that. Their "Crash-Me" test suite shows (at least) that they > > have put a large effort into testing. > > No, it means they've put a crap effort into trying to make > other databases look bad... It does not achieve that goal. Most of the criticism leveled at their efforts sound like fearful hand waving to me. True, I have not studied the test as carefully as others have. But the PostgreSQL test is not superior to the MySQL test. I have put considerable effort into the PostgreSQL regression test. We achieved 100% success on the Win32 platform, including dynamic loading of functions.
> I don't have a lot of faith in huge automated test efforts. They're > great at ensuring you don't make the same mistakes you made once before, > but in my experience the nastiest bugs are the ones you haven't seen > before and would never in a million years have dreamed to test for. > Thus, the best test team is a bunch of people doing unplanned things > with the software, on a wide variety of platforms... Which is why I never use a .0 release of PostgreSQL :) Chris
> Things like that drive the credibility of the whole thing to the floor. > Maybe something like this should exist for Postgres, but it's not > crash-me. Maybe the NIST compliance test is adequate. Plus I belive the RedHat people are getting PostgreSQL through the NIST compliance tests at the moment...I'd love to see MySQL pass them... Chris
> Sounds like testing for the existence of a bug. > X = NULL > X <= NULL > X >= NULL > Etc. must always test false, regardless of the contents of X. Test for > equality with NULL is a conformance error if NULL == NULL returns true. They should all return NULL, not false... > > dates like '00-00-0000'); Yes, that's MySQL's idea of a NULL date. In fact, it's exactly what you get when you insert a NULL date into a NOT NULL column - hooray the test proves that MySQL accepts NULL values in NOT NULL columns... Chris
Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes:
>> Maybe the NIST compliance test is adequate.
> Plus I belive the RedHat people are getting PostgreSQL through the NIST
> compliance tests at the moment...I'd love to see MySQL pass them...
FWIW, the first pass of those tests is complete, and it turned up
exactly one bug that we didn't already know of (the
outer-level-aggregate bizarrity that I fixed last week ... which MySQL
wouldn't be subject to since they haven't got subselects ...)
The work is not done, because there are some tests that couldn't be run
because they were blocked by known noncompliances (such as lack of
updatable views). But I'm not getting a sense that we will learn a
whole lot from the NIST tests.
Further details will be published soon...
regards, tom lane
> If you mean the one that comes with PostgreSQL, then I think the MySQL > test is better. The PostgreSQL test seems to focus more on extensions > than anything else. What the? It tests no extensions. The extensions have their own regression tests. > Most of the criticism leveled at their efforts sound like fearful hand > waving to me. True, I have not studied the test as carefully as others > have. But the PostgreSQL test is not superior to the MySQL test. I > have put considerable effort into the PostgreSQL regression test. We > achieved 100% success on the Win32 platform, including dynamic loading > of functions. Notice that it tests absulutely no parallel functionality, whereas PostgreSQL tests things in parallel to check for concurrency problems: "Note that this benchmark is single threaded, so it measures the minimum time for the operations. We plan to in the future add a lot of multi-threaded tests to the benchmark suite. " It's said that for at least 4 years now. Crash-me has nothing to do with testing, it jsut checks to see what features a db supports: "crash-me tries to determine what features a database supports and what its capabilities and limitations are by actually running queries. For example, it determines: What column types are supported How many indexes are supported What functions are supported How big a query can be How big a VARCHAR column can be" Obviously it has nothing to do with can I index every type in the system, can I use the index to look up a set of test values, etc., etc. Chris
"Dann Corbit" <DCorbit@connx.com> writes: > Look at this: > http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on This looks a little cleaner than the last time I looked at it (more than three years ago), but it's still fundamentally a marketing effort. It is not an exercise in spec compliance measurement, because there are hundreds of bullet points that all look exactly alike, whether they are measuring spec-required elements, random vendor extensions to the spec, or spec violations. To take just one example of the latter, "Calculate 1--1" is still shown with a green star for MySQL and a failure for Postgres, when a more correct reading would be "Fails to recognize SQL-standard -- comment syntax" for MySQL. And yes, they were called out on this three years ago, and no they haven't fixed the entry since then. I should believe that there is any good faith on their part? For another example, take a close look at the "Quoting" section, which purports to measure compliance to the spec's ideas about how to quote an identifier. Postgres accepts double-quoted identifiers per spec, including doubled double quotes per spec, and rejects bracketed or backquoted identifiers per spec. MySQL is apparently spec compliant on just one of those four points. Curious that they manage to end up with a better looking display than us in this section; in particular note that Postgres is specifically claimed *not* to handle double-quoted identifiers. (Memory is fuzzy after three years, but IIRC when you look at the actual test code being used, it tests more than whether double quoted identifiers are allowed, and really is failing us on some quite unrelated detail.) Another point worth mentioning is that most of the numerical limits shown in the table have nothing to do with actual server limits, but with random limitations of their test process. For instance, I'm not sure what "max index part length 235328" really means, but I am pretty sure it's got nothing to do with the Postgres server. Or look at "constant string size in SELECT 16777207" ... nope, there's no such limit. (If they'd put a "+" in there then it'd be okay, but no.) I still remember watching crash-me trying to measure the max query length of Postgres 7.0: the crashme client process dumped core before Postgres did, after which the controlling script announced that we weren't crash-safe. > So far, I have seen three problems pointed out (out of 600+ tests). These are the high spots from three-year-old memories. Do you really want a detailed analysis? A quick look at their table recalls plenty of bogosity to my mind. A last point is that this table is comparing MySQL 4.1 (bleeding edge alpha release) against PG 7.2 (one full major release behind the times). While I cannot really blame the MySQL guys for not being up-to-the- minute on everyone else's releases, this does emphasize the key point, namely that this isn't a fair comparison run by disinterested parties but a marketing effort of, by, and for MySQL. regards, tom lane
> -----Original Message----- > From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > Sent: Friday, June 20, 2003 11:47 PM > To: Dann Corbit > Cc: Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > "Dann Corbit" <DCorbit@connx.com> writes: > > Look at this: > > > http://www.mysql.com/information/crash-me.php?mysql_4_1=on&postgres=on > > This looks a little cleaner than the last time I looked at it > (more than three years ago), but it's still fundamentally a > marketing effort. It is not an exercise in spec compliance > measurement, because there are hundreds of bullet points that > all look exactly alike, whether they are measuring > spec-required elements, random vendor extensions to the spec, > or spec violations. To take just one example of the latter, > "Calculate 1--1" is still shown with a green star for MySQL > and a failure for Postgres, when a more correct reading would > be "Fails to recognize SQL-standard -- comment syntax" for > MySQL. And yes, they were called out on this three years > ago, and no they haven't fixed the entry since then. I > should believe that there is any good faith on their part? > > For another example, take a close look at the "Quoting" > section, which purports to measure compliance to the spec's > ideas about how to quote an identifier. Postgres accepts > double-quoted identifiers per spec, including doubled double > quotes per spec, and rejects bracketed or backquoted > identifiers per spec. MySQL is apparently spec compliant on > just one of those four points. Curious that they manage to > end up with a better looking display than us in this section; > in particular note that Postgres is specifically claimed > *not* to handle double-quoted identifiers. (Memory is fuzzy > after three years, but IIRC when you look at the actual test > code being used, it tests more than whether double quoted > identifiers are allowed, and really is failing us on some > quite unrelated detail.) > > Another point worth mentioning is that most of the numerical > limits shown in the table have nothing to do with actual > server limits, but with random limitations of their test > process. For instance, I'm not sure what "max index part > length 235328" really means, but I am pretty sure it's got > nothing to do with the Postgres server. Or look at "constant > string size in SELECT 16777207" ... nope, there's no such > limit. (If they'd put a "+" in there then it'd be okay, but > no.) I still remember watching crash-me trying to measure the > max query length of Postgres 7.0: the crashme client process > dumped core before Postgres did, after which the controlling > script announced that we weren't crash-safe. > > > So far, I have seen three problems pointed out (out of 600+ tests). > > These are the high spots from three-year-old memories. Do > you really want a detailed analysis? A quick look at their > table recalls plenty of bogosity to my mind. > > A last point is that this table is comparing MySQL 4.1 > (bleeding edge alpha release) against PG 7.2 (one full major > release behind the times). While I cannot really blame the > MySQL guys for not being up-to-the- minute on everyone else's > releases, this does emphasize the key point, namely that this > isn't a fair comparison run by disinterested parties but a > marketing effort of, by, and for MySQL. It seems pretty clear that there are warts on the Crashme test. Perhaps 70% or so is truly useful. Maybe the useful subset could be approximated or modified to be useful as a general tool set. Not too surprising that a commercial enterprise tries to bend the facts in their favor a bit. Some other stuff worth note: http://osdb.sourceforge.net/ http://sourceforge.net/projects/osdldbt (looks like someone has put a bunch of PostgreSQL effort into it. http://sourceforge.net/projects/ltp/ (DOTS) http://www.mysql.com/portal/software/item-222.html (I won't mention where it's from) ftp://ftp.cs.wisc.edu/OO7/ Win32 specific, but has source code: http://www.mipt.sw.ru/en/install/ots/ (ODBC testing) http://www.mipt.sw.ru/en/install/ats/ (ADO testing) Some other interesting stuff is found there too... Test tools links: http://www.softwareqatest.com/qattls1.html http://www.aptest.com/resources.html
----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> Cc: "Alvaro Herrera" <alvherre@dcc.uchile.cl>; "Dann Corbit" <DCorbit@connx.com>; "Jason Earl" <jason.earl@simplot.com>; "PostgreSQL-development" <pgsql-hackers@postgresql.org> Sent: Saturday, June 21, 2003 8:33 AM Subject: Re: [HACKERS] Two weeks to feature freeze > Christopher Kings-Lynne <chriskl@familyhealth.com.au> writes: > >> Maybe the NIST compliance test is adequate. > > > Plus I belive the RedHat people are getting PostgreSQL through the NIST > > compliance tests at the moment...I'd love to see MySQL pass them... > > FWIW, the first pass of those tests is complete, and it turned up > exactly one bug that we didn't already know of (the > outer-level-aggregate bizarrity that I fixed last week ... which MySQL > wouldn't be subject to since they haven't got subselects ...) > > The work is not done, because there are some tests that couldn't be run > because they were blocked by known noncompliances (such as lack of > updatable views). But I'm not getting a sense that we will learn a > whole lot from the NIST tests. As far as I can see NIST (http://www.itl.nist.gov/div897/ctg/sql_form.htm) tests are used *only* for testing SQL92 conformance. Latest available test suite version 6, dated 12/1996. Are you using about 6-7 years old test suite? Perhaps I am wrong, I would be happy If someone could point me up-to-date info about NIST conformance testing. regards, Alvis Tunkelis
Dann Corbit writes: > So far, I have seen three problems pointed out (out of 600+ tests). > That's 0.5% defects. Why not just drop the stupid tests, or bend them > to test for what they ought to be testing. The problem with crashme is that it tells you nothing of practical value. It doesn't tell you whether PostgreSQL works right. It doesn't tell you whether PostgreSQL works well. It doesn't tell you whether PostgreSQL conforms to some standard. It doesn't even tell whether PostgreSQL is compatible to some other product. The only thing that you could possibly get out of fixing crashme is to look better in crashme. And for the above reasons, there is little interest in working on that. -- Peter Eisentraut peter_e@gmx.net
Thomas Swan writes: > Have you considered something similar to the Mozilla tinderbox approach > where you have a daemon checkout the cvs, compile, run regression tests, > and report a status or be able to report a status? Even if you could achieve near complete coverage of the platforms, platform versions, and auxilliary software versions and combinations that PostgreSQL runs with, in most cases, something breaks on a new version or combination of these things. -- Peter Eisentraut peter_e@gmx.net
> > PostgreSQL has a comprehensive regression suite that is run > > by the developers all the time... > > If you mean the one that comes with PostgreSQL, then I think the MySQL > test is better. The PostgreSQL test seems to focus more on extensions > than anything else. I would be happy to make additions to the regression tests if you can give me a list of items that are missing. I've looked and didn't see anything obvious aside from inter-connection testing. Yes, tests run in parallel on multiple connections, but there is no interaction between since there is not a method of controlling the timing at the moment. -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
On Fri, 20 Jun 2003, Josh Berkus wrote: > Ultimately, this is one of those "technical" vs. "marketing" questions > ... whether to release now with a bunch of back-end features that the > current users want, or to release later and include the features that we > said were going to be in 7.4. And PostgreSQL is a technical project, > not a marketing one. Technical or Marketing, I think ppl are putting too much emphasis on 'visible features' and not enough on the 'not so visible' ones ... improvements to both performance and footprint are massive changes, but they are more difficult to 'market', then, say, adding schemas was ...
On Fri, 20 Jun 2003, Tom Lane wrote: > Time was that we had a major release every 3 or 4 months. As the > project matures I think it's appropriate for the cycle to get slower: a > lot of low-hanging fruit is gone, so we have larger jobs to tackle, plus > users are using PG for larger databases and don't want to face > major-version changes too often. But I don't want it to get to be a > year on average between releases, at least not yet. 8 or 9 months seems > reasonable, and by that standard we're overdue. Note that with how we've been releasing 'minors' on v7.3.x semi-regularly, slippage isn't *as* big an issue as it could have been ...
On Fri, 20 Jun 2003, Jason Earl wrote: > Heck, there are probably more than 70 machines running > CVS versions of PostgreSQL right this minute (Marc, any download > numbers to back this up?). Unfortunately, most ppl testing would be using CVS or CVSup, which don't (or, at least, I haven't been able to find?) log such .. download wise, through the FTP site, maybe 20 downloads since the 4th of June of the snapshot ...
On Fri, 20 Jun 2003, Dann Corbit wrote: > > Hmm... I must have missed the huge corporation paying for in > > house testing of PostgreSQL. In the Free Software world the > > "beta team" is all of those people that need the new features > > so badly that they are willing to risk their own data and > > hardware testing it. > > I don't see how this model can possibly succeed then. You can't just > hope that your end users will: > 1. Exhaustively test > 2. Accurately report the findings But it does, and has for 10 years now ... > Our beta customers do help us to find bugs. Bugs reported by customers > for released products are extremely rare. Check the past archives for the mailing lists ... our "bugs reported by end users for released products" is extremely rare also, and *generally* is a result of them doing something that nobody had thought to test for before ... > Spoken like a programmer. Yes, real world data *always* turns up things > that neither the testers nor the programmers imagined. But a huge and > comprehensive conformance testing effort will turn up 99% of the > problems. And ours do ... I don't believe I can recall us having a release where we've had a stream of problem reports come flying in afterwards ... we might get one or two from ppl that have hit a 'never before seen' bug, that generally gets fixed very quickly ... > 100% code coverage is impossible. > Program proving is impossible. > 0% defect code delivery is impossible. > > But you should try to approach the ideal as closely as can be attained. And we do ... > The tests are good tests. They cover a wide range of features and > functions and discover if you can cause permanent damage to a database > by simply performing end-user queries. The scripts are a bit hokey, but > it isn't all that difficult to get them to run. Well, if you would like to volunteer to run them against PostgreSQL, and let us know what fails, we can let you know why said test is wrong in the first place ... we've been through crash-me several times before, and 'fixing crash-me' was more work then it was worth ... > > Basically any time a competitor differed from > > MySQL an error would be generated (despite the fact that it > > was very likely that it was MySQL that was wrong). > > This is unfair and untrue. (I have no connection whatsoever with the > MySQL group, BTW). Been there, done that ... even tried to get changes made to make the tests more accurate ... it was like trying to move a mountain ... > PostgreSQL has an excellent programming team. Why not try to recruit a > similar testing team? I think it would strongly differentiate the tool > set from similar free stuff. Are you volunteering? We already have a testing team we're happy with, but if you would like to extend it with your resources, please feel free to join in ...
Peter Eisentraut <peter_e@gmx.net> writes:
> Thomas Swan writes:
>> Have you considered something similar to the Mozilla tinderbox approach
>> where you have a daemon checkout the cvs, compile, run regression tests,
>> and report a status or be able to report a status?
> Even if you could achieve near complete coverage of the platforms,
> platform versions, and auxilliary software versions and combinations that
> PostgreSQL runs with, in most cases, something breaks on a new
> version or combination of these things.
Still, whenever we're doing something that interacts at all with the OS,
it seems we get breakages that don't show in the original author's
testing, but only pop up days to months later when some beta tester
tries the code on platform P or using option Q. The current
difficulties with the IPv6 patches are a fine case in point.
If we could get feedback more easily about whether a proposed patch
compiles and passes regression on a variety of platforms, we could
reduce the pain involved by a great deal, simply because the problems
could be fixed while the code is still fresh in mind.
I don't think there is any company involved with Postgres that is
willing to commit the resources to run a Mozilla-style tinderbox setup
singlehanded. But I wonder whether we couldn't set up something that is
community-based: get a few dozen people with different platforms to
volunteer to check the code regularly on their own machines. I'm
imagining a cron job that fires daily in the wee hours, pulls the latest
CVS tip, does "make distclean; configure; make; make check", and mails
the results to someplace that puts 'em up on our website.
It's possible that we could adapt the tinderbox software to work this
way, but even if we had to write our own, it seems like a fairly simple
task. And it'd give *much* better feedback on porting problems than we
have now. Sure, there will always be corner cases you don't catch,
but the first rule of testing is the sooner you find a bug the cheaper
it is to fix.
regards, tom lane
--On Saturday, June 21, 2003 11:43:17 -0400 Tom Lane <tgl@sss.pgh.pa.us> wrote: > Peter Eisentraut <peter_e@gmx.net> writes: >> Thomas Swan writes: >>> Have you considered something similar to the Mozilla tinderbox approach >>> where you have a daemon checkout the cvs, compile, run regression tests, >>> and report a status or be able to report a status? > >> Even if you could achieve near complete coverage of the platforms, >> platform versions, and auxilliary software versions and combinations that >> PostgreSQL runs with, in most cases, something breaks on a new >> version or combination of these things. > > Still, whenever we're doing something that interacts at all with the OS, > it seems we get breakages that don't show in the original author's > testing, but only pop up days to months later when some beta tester > tries the code on platform P or using option Q. The current > difficulties with the IPv6 patches are a fine case in point. > If we could get feedback more easily about whether a proposed patch > compiles and passes regression on a variety of platforms, we could > reduce the pain involved by a great deal, simply because the problems > could be fixed while the code is still fresh in mind. > > I don't think there is any company involved with Postgres that is > willing to commit the resources to run a Mozilla-style tinderbox setup > singlehanded. But I wonder whether we couldn't set up something that is > community-based: get a few dozen people with different platforms to > volunteer to check the code regularly on their own machines. I'm > imagining a cron job that fires daily in the wee hours, pulls the latest > CVS tip, does "make distclean; configure; make; make check", and mails > the results to someplace that puts 'em up on our website. > > It's possible that we could adapt the tinderbox software to work this > way, but even if we had to write our own, it seems like a fairly simple > task. And it'd give *much* better feedback on porting problems than we > have now. Sure, there will always be corner cases you don't catch, > but the first rule of testing is the sooner you find a bug the cheaper > it is to fix. > > regards, tom lane > > ---------------------------(end of broadcast)--------------------------- > TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > I'm willing to run such a job on UnixWare 7.1.3 and OpenUnix 8, as well as FreeBSD 4.8 -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 972-414-9812 E-Mail: ler@lerctr.org US Mail: 1905 Steamboat Springs Drive, Garland, TX 75044-6749
<alvis@piladzi-2.biz> writes: > As far as I can see NIST (http://www.itl.nist.gov/div897/ctg/sql_form.htm) > tests are used *only* for testing SQL92 conformance. > Latest available test suite version 6, dated 12/1996. > Are you using about 6-7 years old test suite? > Perhaps I am wrong, I would be happy If someone could point me up-to-date > info about NIST conformance testing. AFAIK, the NIST abandoned that project years ago, so there isn't any more-up-to-date test suite available. Not sure this is a big problem, since SQL92 is not a moving target. It'd be nice if they had kept working and developed tests for the non-entry-level features, but they didn't. regards, tom lane
On Fri, Jun 20, 2003 at 10:04:09PM -0700, Dann Corbit wrote: > > On Fri, Jun 20, 2003 at 09:25:08PM -0700, Dann Corbit wrote: > > > > Citing Tom Lane: > > > > I have been through crash-me in some detail, and it left a > > > > very bad taste in my mouth. Don't bother holding it up as an > > > > example of good practice. > > > > > > Every single test in their list is interesting and useful. > > > > At least on the version I just saw there are several results > > with Postgres that are weird (table names > 500 chars?). > > It does get silly at a point, but I have seen systems with 128 > characters for table names, column names, etc. Some people seem to like > it. Not me. Too much typing. I meant that the real limit on 7.2 was much lower than that unless they twiddled with sources at compile time (there's no configure switch for that AFAIR). Maybe 31 or 63 chars, I don't remember. Do you really trust the rest of the test seeing that they came up with a clearly wrong answer in such a simple test? They can't even "make vacuum run reliably" on 7.1. See the performance test. Maybe they want to test 7.3 with lazy vacuum in place. Why don't they do that? 7.1 is already 2 years old. > > Other things tested are clearly wrong (things that are = > > NULL, > > Sounds like testing for the existence of a bug. > X = NULL > X <= NULL > X >= NULL > Etc. must always test false, regardless of the contents of X. Test for > equality with NULL is a conformance error if NULL == NULL returns true. You see, you are saying "sounds like they are testing". What does the code actually test? Which is the right behaviour? Which behaviour gets the green point, MySQL's or the right one? There are lots of things like this; I don't want to waste my time actually reading the code to see what the correct answer for each test is. About the 1--1 thing Tom mentioned: be aware that Postgres happily accepts the correct 1 - -1 expression, but also correctly fails to "calculate" 1--1. Which one gets the green point? Of course it's the non-compliant one. Also they don't test things they don't support. Is there a test for subselects? What about concurrency? Transactional issues? What about performance when they have their "transaction support" enabled? > So far, I have seen three problems pointed out (out of 600+ tests). > That's 0.5% defects. Why not just drop the stupid tests, or bend them > to test for what they ought to be testing. There's already a mechanism for testing inside Postgres. Maybe more tests are needed, but crash-me offers no real value. I just became aware that the NIST test suite is quite old. Maybe what's needed is to expand it to SQL3 to develop a way of measuring the compliance level. But the cost of doing that is probably prohibitive. -- Alvaro Herrera (<alvherre[a]dcc.uchile.cl>) "Hay quien adquiere la mala costumbre de ser infeliz" (M. A. Evans)
Larry Rosenman wrote: > > > --On Saturday, June 21, 2003 11:43:17 -0400 Tom Lane > <tgl@sss.pgh.pa.us> wrote: > >> Peter Eisentraut <peter_e@gmx.net> writes: >> >>> Thomas Swan writes: >>> >>>> Have you considered something similar to the Mozilla tinderbox >>>> approach >>>> where you have a daemon checkout the cvs, compile, run regression >>>> tests, >>>> and report a status or be able to report a status? >>> >> >>> Even if you could achieve near complete coverage of the platforms, >>> platform versions, and auxilliary software versions and combinations >>> that >>> PostgreSQL runs with, in most cases, something breaks on a new >>> version or combination of these things. >> >> >> Still, whenever we're doing something that interacts at all with the OS, >> it seems we get breakages that don't show in the original author's >> testing, but only pop up days to months later when some beta tester >> tries the code on platform P or using option Q. The current >> difficulties with the IPv6 patches are a fine case in point. >> If we could get feedback more easily about whether a proposed patch >> compiles and passes regression on a variety of platforms, we could >> reduce the pain involved by a great deal, simply because the problems >> could be fixed while the code is still fresh in mind. >> >> I don't think there is any company involved with Postgres that is >> willing to commit the resources to run a Mozilla-style tinderbox setup >> singlehanded. But I wonder whether we couldn't set up something that is >> community-based: get a few dozen people with different platforms to >> volunteer to check the code regularly on their own machines. I'm >> imagining a cron job that fires daily in the wee hours, pulls the latest >> CVS tip, does "make distclean; configure; make; make check", and mails >> the results to someplace that puts 'em up on our website. >> >> It's possible that we could adapt the tinderbox software to work this >> way, but even if we had to write our own, it seems like a fairly simple >> task. And it'd give *much* better feedback on porting problems than we >> have now. Sure, there will always be corner cases you don't catch, >> but the first rule of testing is the sooner you find a bug the cheaper >> it is to fix. >> >> regards, tom lane >> >> ---------------------------(end of broadcast)--------------------------- >> TIP 2: you can get off all lists at once with the unregister command >> (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) >> > > I'm willing to run such a job on UnixWare 7.1.3 and OpenUnix 8, as well > as FreeBSD 4.8 > > > I'll have a machine shortly where I can run RH9 SMP tests..
Tom Lane wrote: > I have been through crash-me in some detail, and it left a very bad > taste in my mouth. Don't bother holding it up as an example of good > practice. You seem to miss Dan's point. The specific implementation of crashme is undoubtedly flawed in a number of ways, but the idea is very useful as part of an acceptance testing suite. In short, it would probably be beneficial to us to fix crashme so that it tests the proper, standards-compliant things and reports the actual results, and then include it in the test suite. Indeed, we could even go so far as to use it for our own marketing purposes! Have it cite, for each test, which part of the SQL spec it's testing and what the result should be. -- Kevin Brown kevin@sysexperts.com
I wrote: > Tom Lane wrote: > > I have been through crash-me in some detail, and it left a very bad > > taste in my mouth. Don't bother holding it up as an example of good > > practice. > > You seem to miss Dan's point. The specific implementation of crashme > is undoubtedly flawed in a number of ways, but the idea is very useful > as part of an acceptance testing suite. In short, it would probably > be beneficial to us to fix crashme so that it tests the proper, > standards-compliant things and reports the actual results, and then > include it in the test suite. Actually, now that I think about it, it would probably be more beneficial to merge any correct tests that we aren't already performing into our existing regression test framework, provided that the end result doesn't take too long to run (as you pointed out elsewhere, regression tests that take a really long time to run simply won't be run by most people, except perhaps in a tinderbox type of environment). Overall, it might be of some benefit to mark individual regression tests with a priority, and then make it possible to run only those tests of a specified priority or higher. That way, the indvidual developer may decide for himself which group of regression tests to run based on the amount of time he's willing to let it take and how much hardware he has to throw at it. And at the same time, it would make it easier for new tests to be included in the suite without worrying about the impact it would have on people running the tests. -- Kevin Brown kevin@sysexperts.com
----- Original Message ----- From: "Tom Lane" <tgl@sss.pgh.pa.us> To: "Peter Eisentraut" <peter_e@gmx.net> Cc: "Thomas Swan" <tswan@idigx.com>; <pgsql-hackers@postgresql.org> Sent: Saturday, June 21, 2003 11:43 AM Subject: Re: [HACKERS] Two weeks to feature freeze > I don't think there is any company involved with Postgres that is > willing to commit the resources to run a Mozilla-style tinderbox setup > singlehanded. But I wonder whether we couldn't set up something that is > community-based: get a few dozen people with different platforms to > volunteer to check the code regularly on their own machines. I'm > imagining a cron job that fires daily in the wee hours, pulls the latest > CVS tip, does "make distclean; configure; make; make check", and mails > the results to someplace that puts 'em up on our website. > > It's possible that we could adapt the tinderbox software to work this > way, but even if we had to write our own, it seems like a fairly simple > task. And it'd give *much* better feedback on porting problems than we > have now. Sure, there will always be corner cases you don't catch, > but the first rule of testing is the sooner you find a bug the cheaper > it is to fix. > Two thoughts: 1. we'd need a matrix of hardware / (OS/version) / other environmental things to ensure some sort of good coverage. 2. we'd need to test various configuration sets too, e.g. --with-krb5 I too have an old spare x86 machine lying around that I can set up with whatever free *nix might not have coverage and contribute to the effort. andrew
Rod Taylor wrote: > > Do we have any "killer" features added to 7.4 that we can shout about? > > There's usually been one or two in the past...? > > A quick glance at the TODO list shows a number of speed improvements in > specific areas (IN, GROUP BY, Subselects in views), ARRAY improvements, > some utility command improvements / additions, and a significant > protocol update. > > The protocol update may not be flashy, but it is a large step forward in > presenting a clean experience for developers using PostgreSQL (reduces > chance of rare, unexpected, and difficult to find logic errors). > > If nothing else, it makes for an excellent cleanup release that rounds > off some of the sharp corners (tab completion for schema elements in > psql, schema dump in psql, fixed cluster support, transactional > truncate, alter sequence, new regex code for fast MultiByte, etc). The problem with cleanup releases is that most of our recent releases have been of that type. Each release is a good step forward, but I was hoping for a set of killer features for this release. Tom said that our low-hanging fruit is gone and only hard items are left. This is certainly true. What is hard to accept is that those big items take _weeks_ of focused development, and we just don't have enough full-time developers who can spend that amount of time to do them. The sad truth is that there is alway something _else_ to do, rather than block out weeks to code a complex feature. And these are usually features that can't be done incrementally, but require a huge input of time before there is any payback. I tried with Win32, and spent a few weeks getting us closer, but my other work of housecleaning (email/patches/cleanup), and marketing (speaking and tutorial preparation) just make it impossible to spend the time needed to complete a big item. And people were rightly upset that the patches weren't getting applied or cleanup done in a timely manner. It is depressing. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Tom Lane wrote: > "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes: > > What about the nested transaction stuff? > > With all due respect to Alvaro et al, I can't imagine that that will > make it into 7.4. (I have no confidence that PITR or Win32 native port > will make it either...) > > > Do we have any "killer" features added to 7.4 that we can shout about? > > We have a lot of pretty good stuff. You're not happy that the > performance of IN (subselect) has been fixed? That btree index bloat is > fixed (at least in large part, it remains to be seen whether the field > performance is all that we need...)? > > In my opinion the project is not at a state where whizzy new Features > with a capital F are going to jump out of the woodwork. We are making > good advances in performance, reliability, SQL spec compliance, and > stuff like that, but fancy-sounding bullet points are hard to come by. What does bother me is that we weren't getting any closer on those _hard_ items. At least with this release, we will be _closer_ on Win32 and PITR. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Oleg Bartunov wrote: > > Do we have any "killer" features added to 7.4 that we can shout about? > > There's usually been one or two in the past...? > > I'm not sure if contrib/tsearch is a "killer" feature, but we hope > to submit completely new version of tsearch V2 before July 1. > Actually, we have stable code already used in some projects but > currently lacking documentation. Several people are working on tutorial, > reference guide. The problem is that Bruce seems is very overloaded and > for sure he'll have many patches close to July 1. Is it possible > to get rights to commit our changes ? I am sorry there has been such a delay in patches. I will try go improve that, or someone else can apply them. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Andrew Dunstan wrote: > > Maybe a better strategy would be to get a release out soon but not wait 6 > months for another release which would contain the Win32 port and the PITR > stuff (assuming those aren't done in time for this release). What concerns me is that we thought that after 7.3, and didn't do much work on either until we got near 7.4 release. I wonder if this is just going to be a pattern, where these items are so large, we can't get any motivation to focus on them until we get near the final release. I guess if each final release gets us a little closer, eventually we will get there, but this process is not ideal. The big puzzle is how do you get people (including myself) motivated to work on a feature that takes a _huge_ amount of work to see any payoff? I would like to know. Anyone? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Jason Earl wrote: > > I'd rather see the dev cycle shortened by a month, then extended ... > > Why couldn't you just release the win32 version of 7.4 when it was > finished. If it takes an extra month then that just gives you guys > the chance to circulate *two* press releases. The Native Win32 port > is likely to make a big enough splash all by itself. I am working to try to get fork/exec and signals in before the feature freeze so a Win32 patch to 7.4 is possible. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sat, 21 Jun 2003, Bruce Momjian wrote: > What does bother me is that we weren't getting any closer on those > _hard_ items. At least with this release, we will be _closer_ on Win32 > and PITR. Maybe our problem is such a ... hatred of #ifdef? Maybe its time to go back a bit to our roots ... get the 'experimental features' in with #ifdef so that others have a chance to look at and work on it, and once ready for prime time, pull the #ifdef's out ... ?
Dann Corbit wrote: > > Adding a new platform--especially a platform as diverse from > > the rest of PostgreSQL's supported platforms as Windows--is > > what adds the work. Testing the new platform is relatively > > easy. All you need to do is to start using the Win32 version > > with real live data. > > That is not testing. Using the world as your beta team seems to be a > business model used by a few software giants that is largely frowned > upon. I would think that there is an opportunity to do things > differently. [Read 'properly']. > > We (at CONNX Solutions Inc.) have a formal release procedure that > includes many tens of thousands of automated tests using dozens of > different platforms. There are literally dozens of machines (I would > guess 70 or so total) running around the clock for 7 days before we even > know if we have a release candidate. The QA team is distinct from the > development team, and if they say "FLOP!" the release goes nowhere. No > formal release until QA passes it. > > If there is no procedure for PostgreSQL of this nature, then there > really needs to be. I am sure that MySQL must have something in place > like that. Their "Crash-Me" test suite shows (at least) that they have > put a large effort into testing. One thing you might be missing is that we have a _very_ close relationship with our users. We can send out code and debug/fix things much faster than a company can that ships binaries. If you look at the changes that go into minor releases (post X.X.0 releases) you will see very few fixes, and the ones we do fine are usually for very esoteric problem cases. Maybe it isn't ideal, but given our limited resources, it works very well. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Dann Corbit wrote: > Perhaps all that is needed is some sort of automated, formal reporting > procedure. For example, a large test set might be created that runs a > thorough regression feature list. When the test completes, a data file > is emailed to some central repository, parsed, and stored in a database. I do have an automated build/initdb/regression that I run every night and email the results to myself. [ "X$1" != "X-n" ] && CLEAN="-c" && shift. /etc/profilepgstoprm -r /u/pg/data# return command error value(pgmakeall $CLEAN2>&1; echo "$?" > $TMP/ret) | (tee $TMP/0; exit `cat $TMP/ret`) &&aspg initdb &&pgstart &&newdb test &&cd /pg/test/regress&&gmake clean &&aspg gmake installcheckgrep warning $TMP/0 | grep -v setproctitle | grep -v find_rule| grep -v yy_flex_realloc | grep -v '\[javac\] 1 warning' I also run this after I apply patches. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Dann Corbit wrote: > That is the worst possible test plan. It totally lacks organization and > there is no hint to define when the feature set has been covered. Ad > hoc testing is a useful addition, but it cannot replace all the standard > tests that have been used by the industry for decades. > > If you run literally hundreds of tests designed to ensure that your > product conforms to ANSI/ISO standards then the bugs that are missed > will be few and far between. Unless you are bad at designing tests. > > Designing tests is busywork. Desiging tests is boring. Nobody wants to > design tests, let alone interpret the results and define correct > baselines. But testing is very, very important. I remember when I was with Great Bridge they said, "Oh, we are going to have a test setup and do all sorts of testing to improve PostgreSQL." I told them I doubted their testing was going to shake out many more bugs than our existing testing setup, and you know what, I was pretty much right. Sure, they found a few, but it wasn't much. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
The Hermit Hacker wrote: > On Sat, 21 Jun 2003, Bruce Momjian wrote: > > > What does bother me is that we weren't getting any closer on those > > _hard_ items. At least with this release, we will be _closer_ on Win32 > > and PITR. > > Maybe our problem is such a ... hatred of #ifdef? Maybe its time to go > back a bit to our roots ... get the 'experimental features' in with #ifdef > so that others have a chance to look at and work on it, and once ready for > prime time, pull the #ifdef's out ... ? That's a tough call. I do worry about readability. We have made Win32 changes, and they aren't ifdefs, and we still have a running system, and I think we can do that for PITR too. I think the big issue, which may be your point, is to get incremental work into CVS as soon as possible so we continue to take small steps. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
I have added a cleaned up version of this to CVS as src/tools/pgtest. --------------------------------------------------------------------------- Bruce Momjian wrote: > Dann Corbit wrote: > > Perhaps all that is needed is some sort of automated, formal reporting > > procedure. For example, a large test set might be created that runs a > > thorough regression feature list. When the test completes, a data file > > is emailed to some central repository, parsed, and stored in a database. > > I do have an automated build/initdb/regression that I run every night > and email the results to myself. > > > [ "X$1" != "X-n" ] && CLEAN="-c" && shift > > . /etc/profile > pgstop > rm -r /u/pg/data > # return command error value > (pgmakeall $CLEAN 2>&1; echo "$?" > $TMP/ret) | > (tee $TMP/0; exit `cat $TMP/ret`) && > aspg initdb && > pgstart && > newdb test && > cd /pg/test/regress && > gmake clean && > aspg gmake installcheck > grep warning $TMP/0 | > grep -v setproctitle | > grep -v find_rule | > grep -v yy_flex_realloc | > grep -v '\[javac\] 1 warning' > > I also run this after I apply patches. > > -- > Bruce Momjian | http://candle.pha.pa.us > pgman@candle.pha.pa.us | (610) 359-1001 > + If your life is a hard drive, | 13 Roberts Road > + Christ can be your backup. | Newtown Square, Pennsylvania 19073 > > ---------------------------(end of broadcast)--------------------------- > TIP 3: if posting/reading through Usenet, please send an appropriate > subscribe-nomail command to majordomo@postgresql.org so that your > message can get through to the mailing list cleanly > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom said that our low-hanging fruit is gone and only hard items are
> left. This is certainly true. What is hard to accept is that those big
> items take _weeks_ of focused development, and we just don't have enough
> full-time developers who can spend that amount of time to do them. The
> sad truth is that there is alway something _else_ to do, rather than
> block out weeks to code a complex feature. And these are usually
> features that can't be done incrementally, but require a huge input of
> time before there is any payback.
I spent weeks doing hash aggregates, weeks doing IN-subselect
optimization, and am in the middle of many weeks on FE/BE protocol
improvement. I am sorry that you don't see these as killer features
... but they are all things that we desperately needed to do.
regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > Tom said that our low-hanging fruit is gone and only hard items are > > left. This is certainly true. What is hard to accept is that those big > > items take _weeks_ of focused development, and we just don't have enough > > full-time developers who can spend that amount of time to do them. The > > sad truth is that there is alway something _else_ to do, rather than > > block out weeks to code a complex feature. And these are usually > > features that can't be done incrementally, but require a huge input of > > time before there is any payback. > > I spent weeks doing hash aggregates, weeks doing IN-subselect > optimization, and am in the middle of many weeks on FE/BE protocol > improvement. I am sorry that you don't see these as killer features > ... but they are all things that we desperately needed to do. > Yes, I know they are _very_ needed, but they don't increase functionality the way Win32 or PITR would do. Please don't feel I am minimizing these features. If I had to choose, I would choose those features over Win32 or PITR. It is just that I wanted all of them. :-( -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
----- Original Message ----- From: "Bruce Momjian" <pgman@candle.pha.pa.us> > Tom said that our low-hanging fruit is gone and only hard items are > left. This is certainly true. What is hard to accept is that those big > items take _weeks_ of focused development, and we just don't have enough > full-time developers who can spend that amount of time to do them. The > sad truth is that there is alway something _else_ to do, rather than > block out weeks to code a complex feature. And these are usually > features that can't be done incrementally, but require a huge input of > time before there is any payback. > > I tried with Win32, and spent a few weeks getting us closer, but my > other work of housecleaning (email/patches/cleanup), and marketing > (speaking and tutorial preparation) just make it impossible to spend the > time needed to complete a big item. And people were rightly upset that > the patches weren't getting applied or cleanup done in a timely manner. > > It is depressing. I was disappointed that Satoshi Nagayasu's two-phase commit patches seemed to be implicitly rejected by lack of an enthusiastic response by any of the core members. Distributed query (not replication) would have been a very nice feature. It's what separates, in part, Oracle Enterprise Edition from the Standard Edition, and it appeared someone (Satoshi Nagayasu) was more than willing to get the ball rolling. But the flight path bothered some I guess so we got nothin' Mike Mascari mascarm@mascari.com
Tom Lane wrote: > >I spent weeks doing hash aggregates, weeks doing IN-subselect >optimization, and am in the middle of many weeks on FE/BE protocol >improvement. I am sorry that you don't see these as killer features >... but they are all things that we desperately needed to do. > > > For me, the 7.4 enhancements are essential, the join optimizations make the difference between "app works" and "app doesn't work", because queries (that can't be changed) that previously ran for ages for non-obvious reasons now speed up to <<1 second. The planner reached a new level of maturity. I'd recommend continuing enhancement work on pgsql, and if a majority feels that features are so killing the version is bumped up one major. I wouldn't see a yet another platform as a reason for this, rather something that vastly extends the field of operation (2PC was mentioned, maybe PITR). Regards, Andreas
Mike Mascari wrote: > I was disappointed that Satoshi Nagayasu's two-phase commit > patches seemed to be implicitly rejected by lack of an > enthusiastic response by any of the core members. Distributed > query (not replication) would have been a very nice feature. > It's what separates, in part, Oracle Enterprise Edition from the > Standard Edition, and it appeared someone (Satoshi Nagayasu) was > more than willing to get the ball rolling. But the flight path > bothered some I guess so we got nothin' I sure want two-phase commit. I don't remember it as being rejected, and we certainly need it, independent of replication. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Tom Lane wrote: > BTW, I would not approve of a response along the lines of "can't you > #ifdef to the point that there are no code changes in the Unix builds?" > No you can't, unless you want to end up with an unmaintainable mess > of #ifdef spaghetti. The thing that makes this hard is the tradeoff > between making the code readable and maintainable (which requires > sharing as much code as possible across platforms) vs isolating > platform-specific considerations. Programming at this level is not > a science but an art form, and it's very hard to get it right the first > time --- especially when none of us have access to all the platforms > that the code must ultimately work on. Exactly my point and the reason I am doing the entire fork+exec stuff over again. Bruce nagged me endlessly to commit the broken parts I had and fix them later. I never agreed with that philosophy because in my experience the worst workarounds live forever. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Dann Corbit wrote: >> PostgreSQL's regression tests (IMHO) are much better than >> MySQL's crash-me scripts. > > They are less thorough in coverage, but not too bad. Right, we are still missing this test that proves 10,000 CREATE+DROP TABLE will ensure that PostgreSQL is slower than MySQL, if you don't VACUUM the catalog ... > > Here is what I suggest: > > PostgreSQL has an excellent programming team. Why not try to recruit a > similar testing team? I think it would strongly differentiate the tool > set from similar free stuff. > > Perhaps all that is needed is some sort of automated, formal reporting > procedure. For example, a large test set might be created that runs a > thorough regression feature list. When the test completes, a data file > is emailed to some central repository, parsed, and stored in a database. Here is what I suggest: Get started :-) Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Dann Corbit wrote: >> -----Original Message----- >> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] >> >> Are you volunteering to create it? Step right up. > > No. And as an outsider, I rather doubt if any procedures I developed > would be taken very seriously. If such procedures are to be developed, > I suspect that they will have to be developed from within if they are to > be successful. What do you think is the way to become an insider? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Alvaro Herrera wrote: > Also they don't test things they don't support. Is there a test for > subselects? What about concurrency? Transactional issues? What about > performance when they have their "transaction support" enabled? Sure don't they. Like their NUMERIC data type, that can *store* any precision, but when you actually calculate with it, it converts to floating point internally ... that's not only a spec violation, in many countries that's a violation of law if used by a bookkeeping system. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
On Sun, 2003-06-22 at 07:44, Bruce Momjian wrote: > Mike Mascari wrote: > > I was disappointed that Satoshi Nagayasu's two-phase commit > > patches seemed to be implicitly rejected by lack of an > > enthusiastic response by any of the core members. Distributed They weren't ready to be committed at the time, nor are they now. The hardest parts are still to come (resume, forget, etc.). I believe he is still working on the third phase: http://snaga.org/pgsql/ -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
Jan Wieck wrote: > Tom Lane wrote: > > BTW, I would not approve of a response along the lines of "can't you > > #ifdef to the point that there are no code changes in the Unix builds?" > > No you can't, unless you want to end up with an unmaintainable mess > > of #ifdef spaghetti. The thing that makes this hard is the tradeoff > > between making the code readable and maintainable (which requires > > sharing as much code as possible across platforms) vs isolating > > platform-specific considerations. Programming at this level is not > > a science but an art form, and it's very hard to get it right the first > > time --- especially when none of us have access to all the platforms > > that the code must ultimately work on. > > Exactly my point and the reason I am doing the entire fork+exec stuff > over again. Bruce nagged me endlessly to commit the broken parts I had > and fix them later. I never agreed with that philosophy because in my > experience the worst workarounds live forever. I wouldn't say nagging ... I would say NAGGING. :-) -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sun, 22 Jun 2003, Bruce Momjian wrote: > Tom Lane wrote: > > > > I spent weeks doing hash aggregates, weeks doing IN-subselect > > optimization, and am in the middle of many weeks on FE/BE protocol > > improvement. I am sorry that you don't see these as killer features > > ... but they are all things that we desperately needed to do. > > > > Yes, I know they are _very_ needed, but they don't increase > functionality the way Win32 or PITR would do. They don't increase functionality for whom? When someone is comparing PostgreSQL to Oracle, as an example, for consideration in a project, I would think that speed would be one thing that they would consider key 'functionality' in that comparison ... no? I'll never use a Win32 port ... so Tom's work on optimizing queries is more important to me then a Win32 port is ... 'functionality' is completely in 'the eye of the beholder' ...
On Sun, 22 Jun 2003, Bruce Momjian wrote: > Mike Mascari wrote: > > I was disappointed that Satoshi Nagayasu's two-phase commit > > patches seemed to be implicitly rejected by lack of an > > enthusiastic response by any of the core members. Distributed > > query (not replication) would have been a very nice feature. > > It's what separates, in part, Oracle Enterprise Edition from the > > Standard Edition, and it appeared someone (Satoshi Nagayasu) was > > more than willing to get the ball rolling. But the flight path > > bothered some I guess so we got nothin' > > I sure want two-phase commit. I don't remember it as being rejected, > and we certainly need it, independent of replication. I don't recall the patch itself :( Mike, do you recall the date(s) for this? Reasons for rejections?
On Sat, 21 Jun 2003, Bruce Momjian wrote: > That's a tough call. I do worry about readability. We have made Win32 > changes, and they aren't ifdefs, and we still have a running system, and > I think we can do that for PITR too. I think the big issue, which may be > your point, is to get incremental work into CVS as soon as possible so > we continue to take small steps. Exactly ... its gotten to the point that the changes needed are large, so ppl are waiting until its all finished before submitting it ...
On Fri, 20 Jun 2003, Dann Corbit wrote: > Designing tests is busywork. Desiging tests is boring. Nobody wants to > design tests, let alone interpret the results and define correct > baselines. But testing is very, very important. But we do do testing ... we even design testing (in the form of the regression tests) ... we just don't do testing that you personally approve of ... and, from what I've seen so far, you aren't willing to actually put *your* time where your mouth is ... design some tests and submit them to us ... if they are valid, they will get used ... If you feel that crash-me is a valid starting point, start there and see where it takes you ...
On Sat, 21 Jun 2003, Dann Corbit wrote: > It seems pretty clear that there are warts on the Crashme test. > Perhaps 70% or so is truly useful. Maybe the useful subset could be > approximated or modified to be useful as a general tool set. And we all wait with baited breath for you to develop and submit such tests ...
The Hermit Hacker wrote: > On Sun, 22 Jun 2003, Bruce Momjian wrote: > >>Mike Mascari wrote: >> >>>I was disappointed that Satoshi Nagayasu's two-phase commit >>>patches seemed to be implicitly rejected by lack of an >>>enthusiastic response by any of the core members. Distributed >>>query (not replication) would have been a very nice feature. >>>It's what separates, in part, Oracle Enterprise Edition from the >>>Standard Edition, and it appeared someone (Satoshi Nagayasu) was >>>more than willing to get the ball rolling. But the flight path >>>bothered some I guess so we got nothin' >> >>I sure want two-phase commit. I don't remember it as being rejected, >>and we certainly need it, independent of replication. > > I don't recall the patch itself :( > > Mike, do you recall the date(s) for this? Reasons for rejections? I choose my words poorly. A discussion arose regarding the 7.4 protocol changes. I suggested looking forward to allow for a 2PC implementation. Satoshi Nagayasu remarked about the work done on 2PC and posted a link to patches: http://snaga.org/pgsql/pgsql-20021025.tgz The thread was here: http://archives.postgresql.org/pgsql-hackers/2002-11/msg00143.php Various people critiqued the work that had been done - protocol change instead of a purely statement-driven implementation, the use of 2PC for sync. replication, etc. And that was the last (and first, IIRC) post from Satoshi Nagayasu. I was worried that PostgreSQL lost the opportunity to have a 2PC implementation, because no one followed up, allowing it to die on the vine. I have learned from Rod Taylor that lack of posts on hackers doesn't mean lack of work: "They weren't ready to be committed at the time, nor are they now. The hardest parts are still to come (resume, forget, etc.). I believe he is still working on the third phase: http://snaga.org/pgsql/ -- Rod Taylor <rbt@rbt.ca>" So I stand corrected. Mike Mascari mascarm@mascari.com
The Hermit Hacker wrote: > On Fri, 20 Jun 2003, Dann Corbit wrote: > >> Designing tests is busywork. Desiging tests is boring. Nobody wants to >> design tests, let alone interpret the results and define correct >> baselines. But testing is very, very important. > > But we do do testing ... we even design testing (in the form of the > regression tests) ... we just don't do testing that you personally approve > of ... and, from what I've seen so far, you aren't willing to actually put > *your* time where your mouth is ... design some tests and submit them to > us ... if they are valid, they will get used ... > > If you feel that crash-me is a valid starting point, start there and see > where it takes you ... Not that fast! I didn't take the time to check but it wouldn't surprise me if MySQL's crash-me is GPL'd and copyright MySQL AB. That's not an optimal point to start PostgreSQL test code from, is it? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
On Sun, 22 Jun 2003, Jan Wieck wrote: > The Hermit Hacker wrote: > > On Fri, 20 Jun 2003, Dann Corbit wrote: > > > >> Designing tests is busywork. Desiging tests is boring. Nobody wants to > >> design tests, let alone interpret the results and define correct > >> baselines. But testing is very, very important. > > > > But we do do testing ... we even design testing (in the form of the > > regression tests) ... we just don't do testing that you personally approve > > of ... and, from what I've seen so far, you aren't willing to actually put > > *your* time where your mouth is ... design some tests and submit them to > > us ... if they are valid, they will get used ... > > > > If you feel that crash-me is a valid starting point, start there and see > > where it takes you ... > > Not that fast! I didn't take the time to check but it wouldn't surprise > me if MySQL's crash-me is GPL'd and copyright MySQL AB. That's not an > optimal point to start PostgreSQL test code from, is it? I didn't say to copy it, but if the format is what Dann feels is required to be taken seriously, it does give a starting point to work from ... the thing is, as I think it was Tom that pointed out, the crash-me is more a feature tester then anything ... but Dann appears to be stuck on it as the 'be all, end all of testing suites' ...
Bruce Momjian writes: > I have added a cleaned up version of this to CVS as src/tools/pgtest. This seems to be a platform-specific reimplementation of 'make clean; make check'. Why bother? -- Peter Eisentraut peter_e@gmx.net
The Hermit Hacker wrote: > > And, actually, for some reason I hadn't thought of the tsearch as being > another 'INDEX' type ... I crawl back over and be quiet now :) > > Oleg, as far as commits are concerned, I have no problems with extending > the privileges to one of your guys for this, just email me seperately who, > and I'll get it setup ... That would help me too. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Peter Eisentraut wrote: > Bruce Momjian writes: > > > I have added a cleaned up version of this to CVS as src/tools/pgtest. > > This seems to be a platform-specific reimplementation of 'make clean; make > check'. Why bother? Marc requested it. Is there anything platform-specific except the greps? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Let me add my two cents .. I think something like PostgreSQL needs two test suites - an acceptance test (to ensure that checkins don't break functionality) and a regression test suite. What we call the regression test suite is really an acceptance test. Tom Lane is absolutely right in asserting that a test suite that takes a week to run will mean that people won't test at all. Personally, I can (and have for many years) tolerate acceptance test suites that take upto an hour. The existence of such an acceptance test should not however obviate the presence of a wider regression test suite. It should be fine to have an entire suite of regression test buckets take a week to run, because you only start running them once you have a release candidate or equivalent. Of course, setting up a regression test suite takes effort. There is no need, however, to spend umpteen amounts of time writing the buckets. What can be done is to incrementally build it up. So whenever we have a significant new feature, somebody (preferably not the key developers) could take the time to set up a set of test cases that try to test it thoroughly. It's okay if such a test bucket takes 10-15 minutes to run. Then this can get rolled up into the regression suite while a very small "representative" test case makes it to the acceptance test suite. Of course, in the open source world, these things take resources and are not easy to do. I certainly think that the base regression test suite is great. We have clearing the pgsql regression test a checkin requirement for TelegraphCQ developers as our goal is to not break pgsql functionality. -- Pip-pip Sailesh http://www.cs.berkeley.edu/~sailesh
On Sun, 22 Jun 2003, Bruce Momjian wrote: > Peter Eisentraut wrote: > > Bruce Momjian writes: > > > > > I have added a cleaned up version of this to CVS as src/tools/pgtest. > > > > This seems to be a platform-specific reimplementation of 'make clean; make > > check'. Why bother? > > Marc requested it. Is there anything platform-specific except the greps? Ya, the script looked like it did a bit more then just a 'make clean; make check' ... doesn't it?
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I sure want two-phase commit. I don't remember it as being rejected,
> and we certainly need it, independent of replication.
Is 2PC a real-world solution to any real-world problem? I have never
seen a satisfactory explanation of what you do when you've reported that
you're ready to commit and no confirmation ever comes back. Sooner or
later you must violate the protocol in one direction or the other (ie,
commit without confirmation or roll back in violation of your promise
of being able to commit).
I think it's a cool-sounding phrase that does not actually work in
practice.
regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > I sure want two-phase commit. I don't remember it as being rejected, > > and we certainly need it, independent of replication. > > Is 2PC a real-world solution to any real-world problem? I have never > seen a satisfactory explanation of what you do when you've reported that > you're ready to commit and no confirmation ever comes back. Sooner or > later you must violate the protocol in one direction or the other (ie, > commit without confirmation or roll back in violation of your promise > of being able to commit). > > I think it's a cool-sounding phrase that does not actually work in > practice. I think 2PC can be used to build more complex features, like using dblink for cross-db modification queries. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sun, 22 Jun 2003, Bruce Momjian wrote: > Tom Lane wrote: > > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > > I sure want two-phase commit. I don't remember it as being rejected, > > > and we certainly need it, independent of replication. > > > > Is 2PC a real-world solution to any real-world problem? I have never > > seen a satisfactory explanation of what you do when you've reported that > > you're ready to commit and no confirmation ever comes back. Sooner or > > later you must violate the protocol in one direction or the other (ie, > > commit without confirmation or roll back in violation of your promise > > of being able to commit). > > > > I think it's a cool-sounding phrase that does not actually work in > > practice. > > I think 2PC can be used to build more complex features, like using > dblink for cross-db modification queries. That was my understanding too ... as soon as you try and go distributed, you need some method of ensuring that the data is constant across them all ...
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Tom Lane wrote:
>> I think it's a cool-sounding phrase that does not actually work in
>> practice.
> I think 2PC can be used to build more complex features,
Only if it works to begin with. If it's unreliable, what are you gonna
build on it?
regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > Tom Lane wrote: > >> I think it's a cool-sounding phrase that does not actually work in > >> practice. > > > I think 2PC can be used to build more complex features, > > Only if it works to begin with. If it's unreliable, what are you gonna > build on it? The question was whether 2PC is useful. The question wasn't if an unreliable 2PC was useful. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> The question was whether 2PC is useful. The question wasn't if an
> unreliable 2PC was useful.
My question is whether there is such a thing as reliable 2PC. I sure
don't see how you build that.
regards, tom lane
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > The question was whether 2PC is useful. The question wasn't if an > > unreliable 2PC was useful. > > My question is whether there is such a thing as reliable 2PC. I sure > don't see how you build that. Other databases use 2PC --- are you saying none of them are reliable? -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Tom Lane wrote: > Bruce Momjian <pgman@candle.pha.pa.us> writes: >> I sure want two-phase commit. I don't remember it as being rejected, >> and we certainly need it, independent of replication. > > Is 2PC a real-world solution to any real-world problem? I have never > seen a satisfactory explanation of what you do when you've reported that > you're ready to commit and no confirmation ever comes back. Sooner or > later you must violate the protocol in one direction or the other (ie, > commit without confirmation or roll back in violation of your promise > of being able to commit). > > I think it's a cool-sounding phrase that does not actually work in > practice. The other problem I was missing being addressed is what happens if one promised "I can commit" and crashes? Not exactly at the time he crashes, but more at the time he restarts? Doesn't he have to restart into exactly that state of "I can commit", with all locks in place and yet being able to rollback and then again ask "and what now"? I would be surprised if said patch does that ... very *positively* surprised! Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Other databases use 2PC --- are you saying none of them are reliable?
Perhaps they're just smarter than I am. My question stands: what do
you do when the controller doesn't respond after you promise to commit?
Without a believable answer to that, I have no confidence in the idea.
regards, tom lane
>>>>> "Bruce" == Bruce Momjian <pgman@candle.pha.pa.us> writes: Bruce> Tom Lane wrote: >> Bruce Momjian <pgman@candle.pha.pa.us> writes: > The question >> was whether 2PC is useful. The question wasn't if an > >> unreliable 2PC was useful. >> >> My question is whether there is such a thingas reliable 2PC. >> I sure don't see how you build that. Bruce> Other databases use 2PC --- are you saying none of them are Bruce> reliable? And they use them for both federated read/write (what you refer to as distributed access through dblink) and for clustered configurations. I'm not sure if I understand Tom's beef - I think he is concerned about what happens if a subordinate does not respond to a prepare message. I would assume that the co-ordinator would not let the commit go through until it has received confirmations from every subordinate. The application's commit will remain blocked against the co-ordinator when this takes place. That said, I agree that 2PC (and variants) is rather heavy weight when in widely distributed configurations. (Although I guess in practice, many people use Presumed Abort and not vanilla 2PC as PA results in fewer log flushes for read-only transactions.) -- Pip-pip Sailesh http://www.cs.berkeley.edu/~sailesh
Jan Wieck <JanWieck@Yahoo.com> writes:
> The other problem I was missing being addressed is what happens if one
> promised "I can commit" and crashes? Not exactly at the time he crashes,
> but more at the time he restarts? Doesn't he have to restart into
> exactly that state of "I can commit", with all locks in place
Yes, I think he does --- which adds a whole 'nother layer of complexity
and performance penalty to the thing, because all those held locks etc
have to be recorded on disk before you promise to commit.
That part is soluble in theory though, ie, I believe that it can be
done (not efficiently, but it can be done). I don't see what to do
about the no-commit-ack problem.
regards, tom lane
Sailesh Krishnamurthy <sailesh@cs.berkeley.edu> writes:
> I'm not sure if I understand Tom's beef - I think he is concerned
> about what happens if a subordinate does not respond to a prepare
> message. I would assume that the co-ordinator would not let the commit
> go through until it has received confirmations from every
> subordinate.
No. I want to know what the subordinate does when it's promised to
commit and the co-ordinator never responds. AFAICS the subordinate
is screwed --- it can't commit, and it can't abort, and it can't expect
to make progress indefinitely on other work while it's holding locks
for the not-quite-committed transaction.
regards, tom lane
> No. I want to know what the subordinate does when it's promised to > commit and the co-ordinator never responds. AFAICS the subordinate > is screwed --- it can't commit, and it can't abort, and it can't expect > to make progress indefinitely on other work while it's holding locks > for the not-quite-committed transaction. It takes itself offline and you need to resync it later?? Chris
On Sun, 22 Jun 2003, Bruce Momjian wrote: > Tom Lane wrote: > > Bruce Momjian <pgman@candle.pha.pa.us> writes: > > > Tom Lane wrote: > > >> I think it's a cool-sounding phrase that does not actually work in > > >> practice. > > > > > I think 2PC can be used to build more complex features, > > > > Only if it works to begin with. If it's unreliable, what are you gonna > > build on it? > > The question was whether 2PC is useful. The question wasn't if an > unreliable 2PC was useful. I have to back Bruce up on this one ... is there a reason why 2PC couldn't be made reliable? I'm guessin that Oracle supports 2PC ... ? If so, is it unreliable?
On Mon, 23 Jun 2003, Sailesh Krishnamurthy wrote: > I'm not sure if I understand Tom's beef - I think he is concerned about > what happens if a subordinate does not respond to a prepare message. I > would assume that the co-ordinator would not let the commit go through > until it has received confirmations from every subordinate. The > application's commit will remain blocked against the co-ordinator when > this takes place. Wouldn't 2PC have some sort of 'heartbeat' between the co-ordinator and subordinate? Like, if you had multiple subordinates and one crashed, the co-ordinator would have to know that and be able to continue on, no?
On Mon, 23 Jun 2003, Christopher Kings-Lynne wrote: > > No. I want to know what the subordinate does when it's promised to > > commit and the co-ordinator never responds. AFAICS the subordinate > > is screwed --- it can't commit, and it can't abort, and it can't expect > > to make progress indefinitely on other work while it's holding locks > > for the not-quite-committed transaction. > > It takes itself offline and you need to resync it later?? Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the co-ordinator crashes? From the subordinates point of view, it has the complete transaction to commit, but the co-ordinator has gone down without telling it to do so ...
The Hermit Hacker <scrappy@postgresql.org> writes:
> Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the
> co-ordinator crashes?
Or you just lose the network connection for awhile. The worst case
scenario I think is where the co-ordinator got everyone's promise to
commit, and told some of the subordinates to commit, but your own
response gets lost due to network failure. Now what? If you time
out and decide to abort, you're inconsistent with the other
subordinates. On the other hand, you can't commit after a timeout
either, because that loses in the other scenario (where the coordinator
didn't decide to commit). Basically, the subordinate must be willing
to hold its breath *forever*. I don't see how this can work.
regards, tom lane
>>>>> "Tom" == Tom Lane <tgl@sss.pgh.pa.us> writes: Tom> Sailesh Krishnamurthy <sailesh@cs.berkeley.edu> writes: >> I'm not sure if I understand Tom's beef - I think heis >> concerned about what happens if a subordinate does not respond >> to a prepare message. I would assume that theco-ordinator >> would not let the commit go through until it has received >> confirmations from every subordinate. Tom> No. I want to know what the subordinate does when it's Tom> promised to commit and the co-ordinator never responds. Tom> AFAICS the subordinate is screwed --- it can't commit, and it Tom> can't abort, and it can't expect tomake progress Tom> indefinitely on other work while it's holding locks for the Tom> not-quite-committed transaction. Okay I understand what you mean now. AFAIK the general way things happen is that each site has a "recovery procedure" that kicks in after a crash. If the co-ordinator crashes (which could be before or after it sends out COMMIT messages to some of the subordinates), its recovery manager will bring the system up, read the log and ready information about all uncommitted transactions in virtual storage. If a Xact is in the PREPARE stage it will periodically send a message to the co-ordinator asking about what happened to the transaction in question. Once the co-ordinator has come back online it can respond to the query. Of course in the case of a co-ordinator going out of action totally and remaining unconnected this is not a viable solution. If you're making the case that 2PC is not viable on very wide area networks with intermitted connectivity, I agree whole-heartedly. That said, 2PC (and its children, PA and PC) have their place, and are indeed used in many systems. For instance, say you are rigging up a message queueing infrastructure (like MQ-series) to your database (say with NOTIFY), you'd at least like to have the db act as a co-ordinator with the MQ. Or the parallel cluster example I gave earlier. Clustered linux boxes are definitely here although no open-source DBMS offers a parallel solution. -- Pip-pip Sailesh http://www.cs.berkeley.edu/~sailesh
Tom Lane wrote: > The Hermit Hacker <scrappy@postgresql.org> writes: > >>Hrmmm, I see Tom's point (I think!) ... but what if, for instance, the >>co-ordinator crashes? > > Or you just lose the network connection for awhile. The worst case > scenario I think is where the co-ordinator got everyone's promise to > commit, and told some of the subordinates to commit, but your own > response gets lost due to network failure. Now what? If you time > out and decide to abort, you're inconsistent with the other > subordinates. On the other hand, you can't commit after a timeout > either, because that loses in the other scenario (where the coordinator > didn't decide to commit). Basically, the subordinate must be willing > to hold its breath *forever*. Yep. And if the cohort crashes while waiting for the coordinator to come back on-line, if I understand the world correctly, it must be capable of committing the database changes associated with the COMMIT-VOTE response it supplied to the coordinator's PREPARE. It seems this would require REDO? And yet there are thousands of installed distributed databases running enterprises every day. A paper on a "A New Presumed Commit Optimization for Two Phase Commit" describes the cohort as: "If a prepared cohort does not receive a transaction outcome message promptly, or crashes without remembering the outcome, the cohort asks the coordinator for the outcome. It keeps on asking until it gets an answer. (This is the blocking aspect of 2PC.)" I'd just like to point out that: 1) The XA interface defines a 2PC protocol library which allows transaction managers, such as BEAS Tuxedo (and Oracle, for that matter) to use the database in a distributed transaction. Lack of an XA interface for PostgreSQL prohibits its use in major enterprise applications. BEAS Tuxedo can talk to PostgreSQL, but won't allow it to participate in a distributed tx. 2) The users of distributed databases will/should/can know that a cohort will block waiting for the coordinator. We're not talking asynchronous multi-master replication of 4 databases distributed over low-speed communication lines across the country. We're talking about the sales dept. database having a few linked tables to the accounting dept. database, where inserts into the one result in inserts into the other. Mike Mascari mascarm@mascari.com
I wrote: > Tom Lane wrote: > >>Basically, the subordinate must be willing to hold its breath *forever*. > > > Yep. And if the cohort crashes while waiting for the coordinator to > come back on-line, if I understand the world correctly, it must be > capable of committing the database changes associated with the > COMMIT-VOTE response it supplied to the coordinator's PREPARE. It > seems this would require REDO? And yet there are thousands of > installed distributed databases running enterprises every day. Please ignore the REDO remark. It's late where I am... Mike Mascari mascarm@mascari.com
On Sat, 21 Jun 2003, Christopher Kings-Lynne wrote: > Crash-me has nothing to do with testing, it jsut checks to see what > features a db supports: An interesting point is that until recently, crashme said that the postgresql backend crashed on very large queries. The actual problem was that postgresql has NO LIMIT to query size, and the crashme script would keep feeding the postgresql backend larger and larger amounts of query until the internal buffer of the crashme script overran. This failure was attributed to postgresql when it was, in fact a bug in the crashme script. This is not an isolated behaviour of crashme. It's a quick dirty hack job designed to show the differences between MySQL and all the other databases. If it was truly comprehensive (i.e. SQL92 spec testing) there would be hundreds of failure points for MySQL. but it isn't. It tests only those things that are good in MySQL against other databases (for the most part, there is some token effort at including a few things MySQL doesn't do).
On Sat, 21 Jun 2003, Bruce Momjian wrote: > Andrew Dunstan wrote: > > > > Maybe a better strategy would be to get a release out soon but not wait 6 > > months for another release which would contain the Win32 port and the PITR > > stuff (assuming those aren't done in time for this release). > > What concerns me is that we thought that after 7.3, and didn't do much > work on either until we got near 7.4 release. I wonder if this is just > going to be a pattern, where these items are so large, we can't get any > motivation to focus on them until we get near the final release. I > guess if each final release gets us a little closer, eventually we will > get there, but this process is not ideal. > > The big puzzle is how do you get people (including myself) motivated to > work on a feature that takes a _huge_ amount of work to see any payoff? > I would like to know. Anyone? Pizza? :-)
The Hermit Hacker writes: > Ya, the script looked like it did a bit more then just a 'make clean; make > check' ... doesn't it? No. -- Peter Eisentraut peter_e@gmx.net
Peter Eisentraut wrote: > The Hermit Hacker writes: > > > Ya, the script looked like it did a bit more then just a 'make clean; make > > check' ... doesn't it? > > No. Well, it is a nice test template for people who aren't shell script experts, and I have been in the habit of pushing stuff I use into /tools so it is available for others. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Sat, 2003-06-21 at 22:55, Bruce Momjian wrote:
> Andrew Dunstan wrote:
> >
> > Maybe a better strategy would be to get a release out soon but not wait 6
> > months for another release which would contain the Win32 port and the PITR
> > stuff (assuming those aren't done in time for this release).
>
> What concerns me is that we thought that after 7.3, and didn't do much
> work on either until we got near 7.4 release. I wonder if this is just
> going to be a pattern, where these items are so large, we can't get any
> motivation to focus on them until we get near the final release. I
> guess if each final release gets us a little closer, eventually we will
> get there, but this process is not ideal.
>
> The big puzzle is how do you get people (including myself) motivated to
> work on a feature that takes a _huge_ amount of work to see any payoff?
> I would like to know. Anyone?
>
Here's a sure to be wildly unpopular suggestion:
Make the deciding factor for the next release support of "foo" (foo can
be win32, pitr, replication, 2PC, whatever...). This would put ample
pressure on the developers of "foo" to get it done since the whole
release rides on their shoulders. It also might encourage others to help
out more since they can't get something new until "foo" is completed.
This would also prioritize said developers time on the large project
rather than other non-prioritized tasks, and it also helps to ensure a
"killer feature" for those who want such things,
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
On Monday 23 June 2003 10:43 am, Tom Lane wrote: > Robert Treat <xzilla@users.sourceforge.net> writes: > > Here's a sure to be wildly unpopular suggestion: > > > > Make the deciding factor for the next release support of "foo" (foo can > > be win32, pitr, replication, 2PC, whatever...). > > We've done that before (see WAL in 7.1), with unhappy results. well, I did say it would be wildly unpopular ;-) > The > fundamental problem with it is that towards the end of the cycle, > other developers don't know how to schedule their time, because they > don't know when feature freeze is really going to be. People end up > twiddling their thumbs while the schedule slips a few days at a time. > Ok, what if feature freeze comes 1 month after completion of "foo" feature. This way the release is still feature dependent, but people arn't sitting around day by day waiting for foo, and they also don't have to worry about getting caught in the middle of something when foo gets done. (i'm kind of picking 1 month arbitraily, this could be two weeks if that works better). > The target-date-based approach we've taken in the last couple of > releases seems much more productive. > productive on a small scale; for sure. productive for large scale features... well, that's why it's being discussed. Robert Treat
Robert Treat <xzilla@users.sourceforge.net> writes:
> Here's a sure to be wildly unpopular suggestion:
> Make the deciding factor for the next release support of "foo" (foo can
> be win32, pitr, replication, 2PC, whatever...).
We've done that before (see WAL in 7.1), with unhappy results. The
fundamental problem with it is that towards the end of the cycle,
other developers don't know how to schedule their time, because they
don't know when feature freeze is really going to be. People end up
twiddling their thumbs while the schedule slips a few days at a time.
The target-date-based approach we've taken in the last couple of
releases seems much more productive.
regards, tom lane
On Mon, 23 Jun 2003, Bruce Momjian wrote: > Peter Eisentraut wrote: > > The Hermit Hacker writes: > > > > > Ya, the script looked like it did a bit more then just a 'make clean; make > > > check' ... doesn't it? > > > > No. > > Well, it is a nice test template for people who aren't shell script > experts, and I have been in the habit of pushing stuff I use into /tools > so it is available for others. Please keep pushing such scripts out. They're a valuable learning tool for many beginners and a cost little in size to be included.
Scott MArlowe wrote:
> On Sat, 21 Jun 2003, Bruce Momjian wrote:
>
>> The big puzzle is how do you get people (including myself) motivated
>> to work on a feature that takes a _huge_ amount of work to see any
>> payoff? I would like to know. Anyone?
>
> Pizza? :-)
Unfortunately it's off my diet :-(
Seriously, I think an increased willingness to share the work around a bit
would be beneficial. I know that I volunteered to work on the w32 port at a
time when I was, as they say, "between jobs". The response was not
encouraging. Now I am working again and don't have much time available.
I know you can't simply divide tasks with infinite granularity ("nine women
can't make a bay in a month"). Some tasks require one or a few people to get
done and that's all that can be done. But if she/he/they can't get it done,
it's time to send out a call for help, IMNSHO.
Not meaning to criticize - the core team does a great job. I, too, have a
tendency to overcommit and under-delegate. It's very understandable.
andrew
> -----Original Message----- > From: The Hermit Hacker [mailto:scrappy@postgresql.org] > Sent: Sunday, June 22, 2003 12:30 PM > To: Jan Wieck > Cc: The Hermit Hacker; Dann Corbit; Tom Lane; Jason Earl; > PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > On Sun, 22 Jun 2003, Jan Wieck wrote: > > > The Hermit Hacker wrote: > > > On Fri, 20 Jun 2003, Dann Corbit wrote: > > > > > >> Designing tests is busywork. Desiging tests is boring. Nobody > > >> wants to design tests, let alone interpret the results > and define > > >> correct baselines. But testing is very, very important. > > > > > > But we do do testing ... we even design testing (in the > form of the > > > regression tests) ... we just don't do testing that you > personally > > > approve of ... and, from what I've seen so far, you > aren't willing > > > to actually put > > > *your* time where your mouth is ... design some tests and > submit them to > > > us ... if they are valid, they will get used ... > > > > > > If you feel that crash-me is a valid starting point, > start there and > > > see where it takes you ... > > > > Not that fast! I didn't take the time to check but it wouldn't > > surprise me if MySQL's crash-me is GPL'd and copyright MySQL AB. > > That's not an optimal point to start PostgreSQL test code > from, is it? > > I didn't say to copy it, but if the format is what Dann feels > is required to be taken seriously, it does give a starting > point to work from ... > > the thing is, as I think it was Tom that pointed out, the > crash-me is more a feature tester then anything ... but Dann > appears to be stuck on it as the 'be all, end all of testing > suites' ... No. I think it covers a broad spectrum of functionality. It is clear that there are warts in it, and also that it is slanted in a few instances to turn "bugs into features." But I think that a large and thorough test suite that covers all major areas of functionality will prove useful. A test suite that covers just as many features and yet is aimed at honest evaluation would be a big benefit. The larger and more complete a functionality test suite is, the better. If a test suite covers ten times the functionality, it will uncover ten times as many defects. I think it is part of the responsibility of a software vendor to ensure that any released product is as free of defects as possible (even an open source tool set). All real software products larger than a few hundred lines of code have some defects in them. If you are going to trust your companies data to a software tool, you would want it to be as free from defects as is possible to achieve, without rasing the cost prohibitively. I think that performance testing is also a good idea. One of the big benefits of creating a large performance suite is that you can profile your code and find out where the effort is needed to get a big speed increase. I think that the NIST validation suite will be very valuable. The coverage of NIST is pretty good, but that test has warts on it too. You will find (for instance) that there is not one single index built by that test suite. So the joins are all brute force. Yetch. If PostgreSQL can pass all three areas of NIST (SQL, ecpg (the pc directory), and the mc directory) that would be pretty impressive.
> -----Original Message----- > From: Jan Wieck [mailto:JanWieck@Yahoo.com] > Sent: Sunday, June 22, 2003 5:45 AM > To: Dann Corbit > Cc: Tom Lane; Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > Dann Corbit wrote: > >> -----Original Message----- > >> From: Tom Lane [mailto:tgl@sss.pgh.pa.us] > >> > >> Are you volunteering to create it? Step right up. > > > > No. And as an outsider, I rather doubt if any procedures I > developed > > would be taken very seriously. If such procedures are to be > > developed, I suspect that they will have to be developed > from within > > if they are to be successful. > > What do you think is the way to become an insider? Join the CVS tree and make a large and valuable contribution to the project.
Bruce Momjian writes: > Well, it is a nice test template for people who aren't shell script > experts, and I have been in the habit of pushing stuff I use into /tools > so it is available for others. I know and I'm not happy about it. The PostgreSQL source tree isn't a repository of everyone's favorite shell scripts. There's an official method to build and test a PostgreSQL installation. If that is flawed or incomplete, then let's talk about it. But everyone pushing out their own little test methodology without further consideration is not going to help this discussion. -- Peter Eisentraut peter_e@gmx.net
"Dann Corbit" <DCorbit@connx.com> writes:
>> What do you think is the way to become an insider?
> Join the CVS tree and make a large and valuable contribution to the
> project.
For instance, developing an industrial-strength test suite? If you've
got an itch there, scratch it.
regards, tom lane
> -----Original Message----- > From: Bruce Momjian [mailto:pgman@candle.pha.pa.us] > Sent: Saturday, June 21, 2003 8:50 PM > To: Dann Corbit > Cc: Tom Lane; Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > Dann Corbit wrote: > > That is the worst possible test plan. It totally lacks > organization > > and there is no hint to define when the feature set has > been covered. > > Ad hoc testing is a useful addition, but it cannot replace all the > > standard tests that have been used by the industry for decades. > > > > If you run literally hundreds of tests designed to ensure that your > > product conforms to ANSI/ISO standards then the bugs that > are missed > > will be few and far between. Unless you are bad at designing tests. > > > > Designing tests is busywork. Desiging tests is boring. > Nobody wants > > to design tests, let alone interpret the results and define correct > > baselines. But testing is very, very important. > > I remember when I was with Great Bridge they said, "Oh, we > are going to have a test setup and do all sorts of testing to > improve PostgreSQL." I told them I doubted their testing was > going to shake out many more bugs than our existing testing > setup, and you know what, I was pretty much right. Sure, > they found a few, but it wasn't much. PostgreSQL is a fairly mature product, having been in existence in one form or another for many years now. I expect that most of the bugs that surface will be in areas of new functionality. Great Bridge had the right idea though. Let's suppose that they ran 10,000 tests and turned up only one bug. That would be just as valuable (if not more so) than turning up 100 bugs. A large, carefully designed test system is *proof* of software quality, or at least of the effort to determine the quality level. It is also proof of the responsibility of the software's originators. Scenario: You are going to install a tool that your organization will invest its future in. Vendor A: "We think our tool is pretty solid and our end users hardly ever turn up any bugs." Vendor B:" We think our tool is pretty solid and our 8500 tests currently show only 3 defects with the released version, and these are low impact issues. To view our current database of issues, log onto web form <page>." Which tool would you prefer to install?
Peter Eisentraut wrote: > Bruce Momjian writes: > > > Well, it is a nice test template for people who aren't shell script > > experts, and I have been in the habit of pushing stuff I use into /tools > > so it is available for others. > > I know and I'm not happy about it. The PostgreSQL source tree isn't a > repository of everyone's favorite shell scripts. There's an official > method to build and test a PostgreSQL installation. If that is flawed or > incomplete, then let's talk about it. But everyone pushing out their own > little test methodology without further consideration is not going to help > this discussion. I put stuff in /tools so if something happens to me, you guys can keep going. Do I have to be clearer that that? I have RELEASE_CHANGES, which Tom used for 7.3.X releases, pgindent, stuff for finding missing/extraneous includes, static requirements, stuff like that. Unless you can find someone else who agrees with you, it stays. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Dann Corbit wrote: > PostgreSQL is a fairly mature product, having been in existence in one > form or another for many years now. > > I expect that most of the bugs that surface will be in areas of new > functionality. > > Great Bridge had the right idea though. Let's suppose that they ran > 10,000 tests and turned up only one bug. That would be just as valuable > (if not more so) than turning up 100 bugs. A large, carefully designed > test system is *proof* of software quality, or at least of the effort to > determine the quality level. It is also proof of the responsibility of > the software's originators. Look at the cost/benefit ratio to that. If you think we don't have to care about cost/benefit, well, it would be pretty amazing if we didn't. > Scenario: > You are going to install a tool that your organization will invest its > future in. > > Vendor A: "We think our tool is pretty solid and our end users hardly > ever turn up any bugs." > > Vendor B:" We think our tool is pretty solid and our 8500 tests > currently show only 3 defects with the released version, and these are > low impact issues. To view our current database of issues, log onto web > form <page>." > > Which tool would you prefer to install? I don't think commerical vendors, with those 8500 test, are are doing any better in reliability than PostgreSQL, and in fact, I think they are doing worse, and have to expend much more effort than we do. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian writes: > I put stuff in /tools so if something happens to me, you guys can keep > going. Yes, we keep going with make clean; make check, like everyone else. Why don't you consider using that? -- Peter Eisentraut peter_e@gmx.net
On Mon, 23 Jun 2003, Dann Corbit wrote: > Vendor A: "We think our tool is pretty solid and our end users hardly > ever turn up any bugs." > > Vendor B:" We think our tool is pretty solid and our 8500 tests > currently show only 3 defects with the released version, and these are > low impact issues. To view our current database of issues, log onto web > form <page>." > > Which tool would you prefer to install? The one I've tested and found to meet my needs, both now and by providing fixes when I needed it. Real world example: We run Crystal Reports Enterprise edition where I work. It's tested thouroughly (supposedly) and has all kinds of QA. However, getting it to work right and stay up is a nightmare. It's taken them almost a year to get around to testing against the OpenLDAP LDAP server we use. The box said "LDAP V3 compliant" and they assured us that it was. Well, it doesn't work with our LDAP V3 compliant LDAP server at all, and the problem is something they can't fix for months because it doesn't fit into their test cycle. Real world example: Postgresql aggregates in subselects. Someone found a bug in subselects in Postgresql with inner references to outter aggregates. The postgresql team delivered a patch in less than a week. User tested it and it works. I'm not against testing and all, but as one of the many beta testers for Postgresql, I do feel a bit insulted by your attitude that only a cohesive, organized testing effort can result in a reliable product. I take my support of Postgresql seriously, and answer many questions every week in the general, sql, and performance mailing lists. I'm not paid to do it, I stay at work an extra hour or so each day to "pay for it." I test every beta and RC release on our dataset at work, and with a test load to make sure it works for us, and it does. I offered to beta test for Crystal Reports and was told they weren't interested, they can do it in house. Their support, like many big commercial houses, is designed more to make my boss's boss happy, not me, and it shows every day in how they fail to provide timely support for their product while playing CYA to the higherups.
On Mon, 23 Jun 2003, Bruce Momjian wrote: > Peter Eisentraut wrote: > > Bruce Momjian writes: > > > > > Well, it is a nice test template for people who aren't shell script > > > experts, and I have been in the habit of pushing stuff I use into /tools > > > so it is available for others. > > > > I know and I'm not happy about it. The PostgreSQL source tree isn't a > > repository of everyone's favorite shell scripts. There's an official > > method to build and test a PostgreSQL installation. If that is flawed or > > incomplete, then let's talk about it. But everyone pushing out their own > > little test methodology without further consideration is not going to help > > this discussion. > > I put stuff in /tools so if something happens to me, you guys can keep > going. Do I have to be clearer that that? I have RELEASE_CHANGES, > which Tom used for 7.3.X releases, pgindent, stuff for finding > missing/extraneous includes, static requirements, stuff like that. > > Unless you can find someone else who agrees with you, it stays. Peter is coming off awfully paternalistic here. I'd rather have a few extra scripts to look through to find what I need when I'm trying to figure out something than to have a tool that only the hackers know exists and I can only get by asking nicely to see the pretty code. While no one wants to see a contrib or tool directory of a hundred megs, lots of little example files and testing scripts can be nice for nothing else other than the examples they provide. I learned a lot when I first started using pgsql from the things in the contrib directory.
Peter Eisentraut wrote: > Bruce Momjian writes: > > > I put stuff in /tools so if something happens to me, you guys can keep > > going. > > Yes, we keep going with make clean; make check, like everyone else. Why > don't you consider using that? The script is automated to run at night, it captures gmake output while returning the proper error return code, and exits if it finds any problems. There was talk others want to automate nightly builds, so this a start for them. Amazing you find 688 bytes worth discussing. I know you said "what happens if everyone adds their scripts", but something that would be a mess if everyone did it isn't always a proper way to judge if something is appropriate. If someone wants to submit a better test build script, I will merge it into the existing one or replace mine. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Peter Eisentraut wrote: > Bruce Momjian writes: > > > I put stuff in /tools so if something happens to me, you guys can keep > > going. > > Yes, we keep going with make clean; make check, like everyone else. Why > don't you consider using that? Actually, I used to manually do all those tests to test patches, but I found a single script was less error prone, and allowed me to more reliably test patches --- in the old days, I would forget the initdb or regression runs, only to find out later the patch broke something. There is a value to that script for me, and if someone else does patch application, I suggest they use it too. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
> -----Original Message----- > From: scott.marlowe [mailto:scott.marlowe@ihs.com] > Sent: Monday, June 23, 2003 12:25 PM > To: Dann Corbit > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > On Mon, 23 Jun 2003, Dann Corbit wrote: > > > Vendor A: "We think our tool is pretty solid and our end > users hardly > > ever turn up any bugs." > > > > Vendor B:" We think our tool is pretty solid and our 8500 tests > > currently show only 3 defects with the released version, > and these are > > low impact issues. To view our current database of issues, > log onto > > web form <page>." > > > > Which tool would you prefer to install? > > The one I've tested and found to meet my needs, both now and > by providing > fixes when I needed it. > > Real world example: We run Crystal Reports Enterprise > edition where I > work. It's tested thouroughly (supposedly) and has all kinds of QA. > However, getting it to work right and stay up is a nightmare. > It's taken > them almost a year to get around to testing against the OpenLDAP LDAP > server we use. The box said "LDAP V3 compliant" and they > assured us that > it was. Well, it doesn't work with our LDAP V3 compliant > LDAP server at > all, and the problem is something they can't fix for months > because it > doesn't fit into their test cycle. > > > Real world example: Postgresql aggregates in subselects. > Someone found a bug in subselects in Postgresql with inner > references to > outter aggregates. The postgresql team delivered a patch in > less than a > week. User tested it and it works. > > I'm not against testing and all, but as one of the many beta > testers for > Postgresql, I do feel a bit insulted by your attitude that only a > cohesive, organized testing effort can result in a reliable product. Let me rephrase it: "Only a cohesive, organized testing effort can result in a product that is proven reliable." Without such an effort, it is only an educated guess as to whether the product is reliable or not. The data is the most valuable software component in an organization. It is worth more than the hardware and it is worth more than the software. If you are going to trust one billion dollars worth of corporate data on a software system, you ought to ensure that the system has been carefully tested. I don't think that is just an opinion. It's simply common sense. > I take my support of Postgresql seriously, and answer many > questions every > week in the general, sql, and performance mailing lists. I'm > not paid to > do it, I stay at work an extra hour or so each day to "pay > for it." I > test every beta and RC release on our dataset at work, and > with a test > load to make sure it works for us, and it does. > > I offered to beta test for Crystal Reports and was told they weren't > interested, they can do it in house. Their support, like many big > commercial houses, is designed more to make my boss's boss > happy, not me, > and it shows every day in how they fail to provide timely support for > their product while playing CYA to the higherups. A long test cycle does result in a slower patch. But when you get the patch, it is going to work and not introduce new problems. The resistance to testing is typical of programmers. The PostgreSQL group is a group of programmers. I don't think I can change anyone's mind, since the most significant people on the list don't think it is worth the bother. Therefore, I am going to stop harping on it.
scott.marlowe wrote: > Peter is coming off awfully paternalistic here. I'd rather have a few > extra scripts to look through to find what I need when I'm trying to > figure out something than to have a tool that only the hackers know exists > and I can only get by asking nicely to see the pretty code. > > While no one wants to see a contrib or tool directory of a hundred megs, > lots of little example files and testing scripts can be nice for nothing > else other than the examples they provide. I learned a lot when I first > started using pgsql from the things in the contrib directory. Having something that just runs and I don't have to fiddle with it is a big help --- of course, it took me a few years to realize that this is the best way to test patches --- kind of embarassing that I didn't think of it in 1997. I think the patch came out of complaints that my patch application was letting in broken code --- since I have been using it, I am mostly down to weird bugs or platform problem, but the obvious patch problems are pretty much gone, which some people might have noticed. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Dann Corbit wrote: > Let me rephrase it: > "Only a cohesive, organized testing effort can result in a product that > is proven reliable." > > Without such an effort, it is only an educated guess as to whether the > product is reliable or not. The data is the most valuable software > component in an organization. It is worth more than the hardware and it > is worth more than the software. If you are going to trust one billion > dollars worth of corporate data on a software system, you ought to > ensure that the system has been carefully tested. I don't think that is > just an opinion. It's simply common sense. True, it is an "educated guess", but it turns out our educated guess method produces more reliable code than the exhaustive testing method, so though there isn't as much of a _feeling_ of confidence, there is a _result_ of more reliability. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Mon, 23 Jun 2003, Dann Corbit wrote: > > -----Original Message----- > > From: scott.marlowe [mailto:scott.marlowe@ihs.com] > > Sent: Monday, June 23, 2003 12:25 PM > > To: Dann Corbit > > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development > > Subject: Re: [HACKERS] Two weeks to feature freeze > > > > > > On Mon, 23 Jun 2003, Dann Corbit wrote: > > > > > Vendor A: "We think our tool is pretty solid and our end > > users hardly > > > ever turn up any bugs." > > > > > > Vendor B:" We think our tool is pretty solid and our 8500 tests > > > currently show only 3 defects with the released version, > > and these are > > > low impact issues. To view our current database of issues, > > log onto > > > web form <page>." > > > > > > Which tool would you prefer to install? > > > > The one I've tested and found to meet my needs, both now and > > by providing > > fixes when I needed it. > > > > Real world example: We run Crystal Reports Enterprise > > edition where I > > work. It's tested thouroughly (supposedly) and has all kinds of QA. > > However, getting it to work right and stay up is a nightmare. > > It's taken > > them almost a year to get around to testing against the OpenLDAP LDAP > > server we use. The box said "LDAP V3 compliant" and they > > assured us that > > it was. Well, it doesn't work with our LDAP V3 compliant > > LDAP server at > > all, and the problem is something they can't fix for months > > because it > > doesn't fit into their test cycle. > > > > > > Real world example: Postgresql aggregates in subselects. > > Someone found a bug in subselects in Postgresql with inner > > references to > > outter aggregates. The postgresql team delivered a patch in > > less than a > > week. User tested it and it works. > > > > I'm not against testing and all, but as one of the many beta > > testers for > > Postgresql, I do feel a bit insulted by your attitude that only a > > cohesive, organized testing effort can result in a reliable product. > > Let me rephrase it: > "Only a cohesive, organized testing effort can result in a product that > is proven reliable." No, it isn't proven reliable PERIOD, it's proven reliable under the exact conditions of the testing procedure you've implemented. And no matter how idiot proof we try to make Postgresql and its testing, someone else will always make a better idiot. :-) Actually, what I'm saying is that the corner cases are the ones that are hard to predict, and no amount of effort up front is going to find a corner case you haven't thought of yet. > Without such an effort, it is only an educated guess as to whether the > product is reliable or not. The data is the most valuable software > component in an organization. It is worth more than the hardware and it > is worth more than the software. If you are going to trust one billion > dollars worth of corporate data on a software system, you ought to > ensure that the system has been carefully tested. I don't think that is > just an opinion. It's simply common sense. But if that is true, then Postgresql should cause me no end of problems as it crashes down around my feet every other week. Oddly, the dbas for the other systems here at work (Oracle and MSSQL server) have a much higher workload keeping their factory tested databases up and running. In over four years of use, we have had exactly ZERO downtime of postgresql. Carefully testing the system is what I, the DBA of our postgresql servers, do. Only I know how we use the system, so no matter how you or Bruce or Tom might test it, I'll always be able to do something you wouldn't think of, because you're simply not in my shoes. It's not an educated guess that postgresql works for us, it's proven over and over again every single day by the throrough testing and use of every Postgresql user. > > I take my support of Postgresql seriously, and answer many > > questions every > > week in the general, sql, and performance mailing lists. I'm > > not paid to > > do it, I stay at work an extra hour or so each day to "pay > > for it." I > > test every beta and RC release on our dataset at work, and > > with a test > > load to make sure it works for us, and it does. > > > > I offered to beta test for Crystal Reports and was told they weren't > > interested, they can do it in house. Their support, like many big > > commercial houses, is designed more to make my boss's boss > > happy, not me, > > and it shows every day in how they fail to provide timely support for > > their product while playing CYA to the higherups. > > A long test cycle does result in a slower patch. But when you get the > patch, it is going to work and not introduce new problems. Nice theory, but it isn't provable by my experience. While I've put .0 releases of postgresql into production many a times, usually with no issues at all, I never have and never will put a .0 release of Crystal Reports online. They've taught me well not to trust their initial release. > The resistance to testing is typical of programmers. The PostgreSQL > group is a group of programmers. I don't think I can change anyone's > mind, since the most significant people on the list don't think it is > worth the bother. No, I am NOT A postgresql programmer, I am a Postgresql user. I do program, but that's in support of my job as a PHP dev / database admin. I've found myself looking through the postgresql code only an odd half dozen times. I've never NEEDED to hack it, as it seems to just work. > Therefore, I am going to stop harping on it. Good move. You're busily shooting at ghosts right now. While there's always room for more testing, the best way to do it is to roll up your sleaves and write your own tests or add to the regression tests. I've written quite a few load tests in Perl over the years to make sure that postgresql can hold up to the kind of load we tend to throw at it. So have hundreds of other Postgresql users you've never met. Just because no one has a centralized plan and document for testing doesn't mean it isn't getting done, it just means you can't see it all that well. I think of your testing methodology as the equivalent of the old Soviet Union's Five year plans. They were thorough and well documented, but never worked as well as planned. Postgresql is like the free market system. no one's completely in charge, and the more you try to control it the more it tries to be free. Ultimately, the chaos of the free market won out. It may be reassuring to think your product is very well tested before it goes out the door, but it's a false security, proven over and over by commercial products that simply don't work in the field because of problems that the original designers never envisioned, and now that they have a thorough and long drawn out testing cycle, it simply takes longer and longer to get fixes, while providing little, if any, improvement in quality.
On Mon, 23 Jun 2003, Dann Corbit wrote: > > -----Original Message----- > > From: scott.marlowe [mailto:scott.marlowe@ihs.com] > > Sent: Monday, June 23, 2003 12:25 PM > > To: Dann Corbit > > Cc: Bruce Momjian; Tom Lane; Jason Earl; PostgreSQL-development > > Subject: Re: [HACKERS] Two weeks to feature freeze > > > > > > On Mon, 23 Jun 2003, Dann Corbit wrote: > > > > > Vendor A: "We think our tool is pretty solid and our end > > users hardly > > > ever turn up any bugs." > > > > > > Vendor B:" We think our tool is pretty solid and our 8500 tests > > > currently show only 3 defects with the released version, > > and these are > > > low impact issues. To view our current database of issues, > > log onto > > > web form <page>." > > > > > > Which tool would you prefer to install? > > > > The one I've tested and found to meet my needs, both now and > > by providing > > fixes when I needed it. How about the one that doesn't run tests in order to show how much better it is than the competition but to actually test operation? In other words Vendor B has an interest in having the tests pass, what gives you the confidence it just hasn't listed the ones that fail and that the tests that do pass are not just testing something vendor B wants to show it can do? > > Real world example: We run Crystal Reports Enterprise > > edition where I > > work. It's tested thouroughly (supposedly) and has all kinds of QA. > > However, getting it to work right and stay up is a nightmare. > > It's taken > > them almost a year to get around to testing against the OpenLDAP LDAP > > server we use. The box said "LDAP V3 compliant" and they > > assured us that > > it was. Well, it doesn't work with our LDAP V3 compliant > > LDAP server at > > all, and the problem is something they can't fix for months > > because it > > doesn't fit into their test cycle. > > > > > > Real world example: Postgresql aggregates in subselects. > > Someone found a bug in subselects in Postgresql with inner > > references to > > outter aggregates. The postgresql team delivered a patch in > > less than a > > week. User tested it and it works. > > > > I'm not against testing and all, but as one of the many beta > > testers for > > Postgresql, I do feel a bit insulted by your attitude that only a > > cohesive, organized testing effort can result in a reliable product. > > Let me rephrase it: > "Only a cohesive, organized testing effort can result in a product that > is proven reliable." > > Without such an effort, it is only an educated guess as to whether the > product is reliable or not. The data is the most valuable software > component in an organization. It is worth more than the hardware and it > is worth more than the software. If you are going to trust one billion > dollars worth of corporate data on a software system, you ought to > ensure that the system has been carefully tested. I don't think that is > just an opinion. It's simply common sense. So you've never worked on a project where the data is of high value, since in those circumstances the customer is always going to apply their own acceptance testing anyway. If you think that doesn't happen you try sitting through 2 solid days of Y2k testing on _one_ system and tell me customers never do their own testing. > Therefore, I am going to stop harping on it. But there is no need to, as has been mentioned before, if the testing is not upto your level of testing submit something that makes it so. Having said that I do believe you mentioned that you didn't have the time to create something but you would be happy to test it, i.e. test the test. -- Nigel J. Andrews
> -----Original Message----- > From: Nigel J. Andrews [mailto:nandrews@investsystems.co.uk] > Sent: Monday, June 23, 2003 1:30 PM > To: Dann Corbit > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; > PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze [snip] > So you've never worked on a project where the data is of high > value, since in those circumstances the customer is always > going to apply their own acceptance testing anyway. If you > think that doesn't happen you try sitting through 2 solid > days of Y2k testing on _one_ system and tell me customers > never do their own testing. Here's an old copy of my resume. You be the judge: ftp://cap.connx.com/C.A.P.%20Biographies/DannCorbit.htm#_Toc441048186 The burden of reliablity testing must not rest solely on the end user. That constitutes negligence on the part of the software vendor (IMO-YMMV). > > Therefore, I am going to stop harping on it. > > But there is no need to, as has been mentioned before, if the > testing is not upto your level of testing submit something > that makes it so. Having said that I do believe you mentioned > that you didn't have the time to create something but you > would be happy to test it, i.e. test the test. I may or may not have time to work on a software test system for PostgreSQL. I do a lot of PostgreSQL work here, and at some point I think I will have valuable contributions to the project.
On Monday 23 June 2003 15:42, Dann Corbit wrote: > Let me rephrase it: > "Only a cohesive, organized testing effort can result in a product that > is proven reliable." One can never 100% prove reliability without time in the field with real-world data, testing or no testing. I would dare say that there are numerous reliable software packages out there in OSS-land that have never had that sort of testing. But it really hinges on ones definition of proof, no? Furthermore, the beta testers here in hackers are not 'end-users' per se. The people in hackers who test the betas are very valuable as our QA team. PostgreSQL is already proven reliable, to various degrees of reliability, enough to where a PostgreSQL backend was approved as the new .ORG registry. The track record continues, without mathematically rigorous and cohesive testing. Such testing would be useful, of course, by it is not required for our purposes. Those who want rigorous testing can do the rigorous testing. You yourself said that your company has a separate QA team from the development team; OK, organize a rigorous QA team. While you won't stop releases (unless you find showstopper bugs, like those that have been found by our wonderful hackers testers), your input into actually testing PostgreSQL (as opposed to trying to convince someone else to test for you) would be valuable. But you're not going to get me to do it; I do enough testing of a varied nature already. I do this for fun. If you find testing fun, more power to you. :-) -- Lamar Owen WGCR Internet Radio 1 Peter 4:11
On Mon, 23 Jun 2003, Peter Eisentraut wrote: > Bruce Momjian writes: > > > Well, it is a nice test template for people who aren't shell script > > experts, and I have been in the habit of pushing stuff I use into /tools > > so it is available for others. > > I know and I'm not happy about it. The PostgreSQL source tree isn't a > repository of everyone's favorite shell scripts. There's an official > method to build and test a PostgreSQL installation. If that is flawed or > incomplete, then let's talk about it. But everyone pushing out their own > little test methodology without further consideration is not going to help > this discussion. 'K, its flawed and incomplete, let's talk about it :) What I was looking for here was something I could add to cron on a machine that would update the source, do a base configure (or give me the ability to give extra options, as the case may be), build, install and test, and report errors to me via email where applicable ... The idea is that it could be something that ppl could have run nightly, in the wee hours, and only when a problem occurs would they be informed so taht they coudl either fix teh error (ie. out of space), or report it to the list(s) ...
Yes, it does some of that, but I don't think it is safe to do a cvs update in an automated fashion, as least on my machine. --------------------------------------------------------------------------- The Hermit Hacker wrote: > > On Mon, 23 Jun 2003, Peter Eisentraut wrote: > > > Bruce Momjian writes: > > > > > Well, it is a nice test template for people who aren't shell script > > > experts, and I have been in the habit of pushing stuff I use into /tools > > > so it is available for others. > > > > I know and I'm not happy about it. The PostgreSQL source tree isn't a > > repository of everyone's favorite shell scripts. There's an official > > method to build and test a PostgreSQL installation. If that is flawed or > > incomplete, then let's talk about it. But everyone pushing out their own > > little test methodology without further consideration is not going to help > > this discussion. > > 'K, its flawed and incomplete, let's talk about it :) > > What I was looking for here was something I could add to cron on a machine > that would update the source, do a base configure (or give me the ability > to give extra options, as the case may be), build, install and test, and > report errors to me via email where applicable ... > > The idea is that it could be something that ppl could have run nightly, in > the wee hours, and only when a problem occurs would they be informed so > taht they coudl either fix teh error (ie. out of space), or report it to > the list(s) ... > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 9: the planner will ignore your desire to choose an index scan if your > joining column's datatypes do not match > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Mon, 23 Jun 2003, Robert Treat wrote: > > The target-date-based approach we've taken in the last couple of > > releases seems much more productive. > > > > productive on a small scale; for sure. productive for large scale > features... well, that's why it's being discussed. 'K, but if we extend the dev cycle in order to get 'foo' in, how is that better then having 'foo' continue to be developed thru the release and committed in the next cycle? If it takes foo 6 months to develop, I'd rather have the release happen after 4 months as per normal (or close to it) and have 'foo' brought in part way into dev cycle 2 ... at least then, if 'foo' ends up taking 7 months, we aren't delaying even further ... Its not like our dev cycles have 'idle periods' where nothing is happening and we're waiting for a feature to come along ... there is *alot* of changes going on that affect alot of ppl that don't really care about feature 'foo' coming along ...
On Mon, 23 Jun 2003, Dann Corbit wrote: > The resistance to testing is typical of programmers. The PostgreSQL > group is a group of programmers. I don't think I can change anyone's > mind, since the most significant people on the list don't think it is > worth the bother. > > Therefore, I am going to stop harping on it. *rofl* I believe several of us have suggested that we would welcome submissions for improved testing ... so, what, we feel that the test that is done is sufficient, but are willing to accept submissions to improve it, but you aren't willing to spend the time/effort to do so? We might be a bunch of 'typical programmers', but you definitely sound like a typical "I want you to do something to make life easier for me, but am not willing to expend the time or effort to do anyting about it" ...
Thank's Robert, that was probably what Bruce needs to call me every other hour now ... Jan Robert Treat wrote: > On Sat, 2003-06-21 at 22:55, Bruce Momjian wrote: >> Andrew Dunstan wrote: >> > >> > Maybe a better strategy would be to get a release out soon but not wait 6 >> > months for another release which would contain the Win32 port and the PITR >> > stuff (assuming those aren't done in time for this release). >> >> What concerns me is that we thought that after 7.3, and didn't do much >> work on either until we got near 7.4 release. I wonder if this is just >> going to be a pattern, where these items are so large, we can't get any >> motivation to focus on them until we get near the final release. I >> guess if each final release gets us a little closer, eventually we will >> get there, but this process is not ideal. >> >> The big puzzle is how do you get people (including myself) motivated to >> work on a feature that takes a _huge_ amount of work to see any payoff? >> I would like to know. Anyone? >> > > Here's a sure to be wildly unpopular suggestion: > > Make the deciding factor for the next release support of "foo" (foo can > be win32, pitr, replication, 2PC, whatever...). This would put ample > pressure on the developers of "foo" to get it done since the whole > release rides on their shoulders. It also might encourage others to help > out more since they can't get something new until "foo" is completed. > This would also prioritize said developers time on the large project > rather than other non-prioritized tasks, and it also helps to ensure a > "killer feature" for those who want such things, > > Robert Treat -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Dann Corbit wrote: >> -----Original Message----- >> From: Jan Wieck [mailto:JanWieck@Yahoo.com] >> >> What do you think is the way to become an insider? > > Join the CVS tree and make a large and valuable contribution to the > project. Go ahead, let's see it. I have contributed a lot of crap over the years. After some other knowingly good and professional programmers layed hand on it, it turned out to be usefull. This isn't meant 100% ironically. I really don't think the original version of the parallel regression test, just to stay with the topic for this example, was worth the bit's needed for storage. But it was a starting point, others built on. And crappy as it was, it caught a bug - funny, isn't it? So don't be afraid, let's see what you got so far. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
scott.marlowe wrote: > On Mon, 23 Jun 2003, Dann Corbit wrote:> [Dann Corbit wrote a lot]> [...] > It may be reassuring to think your product is very well tested before it > goes out the door, but it's a false security, proven over and over by > commercial products that simply don't work in the field because of > problems that the original designers never envisioned, and now that they > have a thorough and long drawn out testing cycle, it simply takes longer > and longer to get fixes, while providing little, if any, improvement in > quality. Scott, it's worse. It's been back in the early 90's, when we had WfW-3.11 systems with some MS-Word dinosaur, and we just lost 14 days of work because it simply crashed on loading the document. The Microsoft support solution was something that lost all the formatting, indexing and cross references of a structured 250 page concept. I don't remember the exact procedure as my brain cells did overcharge, but the dummy on the hotline really believed that their thoroughly tested software wasn't the problem and that the error lies within our document. That that was a file, written by their thoroughly tested software was a point she really didn't catch. This dumb hotline girl is the type of people, Dann Corbit's test strategy will reassure. Plus maybe a few (unfortunately important but otherwise useless) managers. Other than that, it'll not make the life of the average DBA any better. Big amounts of useless tests just give otherwise clueless people the false impression, the error must be somewhere else. MySQL's crash-me is a perfect example for that. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
> -----Original Message----- > From: Jan Wieck [mailto:JanWieck@Yahoo.com] > Sent: Monday, June 23, 2003 10:10 PM > To: scott.marlowe > Cc: Dann Corbit; Bruce Momjian; Tom Lane; Jason Earl; > PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > scott.marlowe wrote: > > On Mon, 23 Jun 2003, Dann Corbit wrote: > > [Dann Corbit wrote a lot] > > [...] > > It may be reassuring to think your product is very well > tested before > > it > > goes out the door, but it's a false security, proven over > and over by > > commercial products that simply don't work in the field because of > > problems that the original designers never envisioned, and > now that they > > have a thorough and long drawn out testing cycle, it simply > takes longer > > and longer to get fixes, while providing little, if any, > improvement in > > quality. > > Scott, it's worse. > > It's been back in the early 90's, when we had WfW-3.11 > systems with some > MS-Word dinosaur, and we just lost 14 days of work because it simply > crashed on loading the document. The Microsoft support solution was > something that lost all the formatting, indexing and cross > references of > a structured 250 page concept. I don't remember the exact > procedure as > my brain cells did overcharge, but the dummy on the hotline really > believed that their thoroughly tested software wasn't the problem and > that the error lies within our document. That that was a > file, written > by their thoroughly tested software was a point she really > didn't catch. > > This dumb hotline girl is the type of people, Dann Corbit's test > strategy will reassure. Plus maybe a few (unfortunately important but > otherwise useless) managers. Other than that, it'll not make > the life of > the average DBA any better. Big amounts of useless tests just give > otherwise clueless people the false impression, the error must be > somewhere else. MySQL's crash-me is a perfect example for that. Do you really believe that such disasters were the result of careful testing before release? Everyone who thinks a careful test plan and implementation is a bad idea is very, very wrong. IMO-YMMV.
Here is a list of a small sample of the citations available from the ACM on software testing: http://portal.acm.org/citation.cfm?id=581358&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=376180&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=367020&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=308790&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=257668&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=248262&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 http://portal.acm.org/citation.cfm?id=227759&coll=portal&dl=ACM&CFID=657 0092&CFTOKEN=81653602 These articles are careful, scientific studies that have been extensively peer reviewed. These articles indicate that testing is a good idea.
Dann Corbit wrote: >> -----Original Message----- >> From: Jan Wieck [mailto:JanWieck@Yahoo.com] >> Sent: Monday, June 23, 2003 10:10 PM >> To: scott.marlowe >> Cc: Dann Corbit; Bruce Momjian; Tom Lane; Jason Earl; >> PostgreSQL-development >> Subject: Re: [HACKERS] Two weeks to feature freeze >> >> >> scott.marlowe wrote: >> > On Mon, 23 Jun 2003, Dann Corbit wrote: >> > [Dann Corbit wrote a lot] >> > [...] >> > It may be reassuring to think your product is very well >> tested before >> > it >> > goes out the door, but it's a false security, proven over >> and over by >> > commercial products that simply don't work in the field because of >> > problems that the original designers never envisioned, and >> now that they >> > have a thorough and long drawn out testing cycle, it simply >> takes longer >> > and longer to get fixes, while providing little, if any, >> improvement in >> > quality. >> >> Scott, it's worse. >> >> It's been back in the early 90's, when we had WfW-3.11 >> systems with some >> MS-Word dinosaur, and we just lost 14 days of work because it simply >> crashed on loading the document. The Microsoft support solution was >> something that lost all the formatting, indexing and cross >> references of >> a structured 250 page concept. I don't remember the exact >> procedure as >> my brain cells did overcharge, but the dummy on the hotline really >> believed that their thoroughly tested software wasn't the problem and >> that the error lies within our document. That that was a >> file, written >> by their thoroughly tested software was a point she really >> didn't catch. >> >> This dumb hotline girl is the type of people, Dann Corbit's test >> strategy will reassure. Plus maybe a few (unfortunately important but >> otherwise useless) managers. Other than that, it'll not make >> the life of >> the average DBA any better. Big amounts of useless tests just give >> otherwise clueless people the false impression, the error must be >> somewhere else. MySQL's crash-me is a perfect example for that. > > Do you really believe that such disasters were the result of careful > testing before release? > > Everyone who thinks a careful test plan and implementation is a bad idea > is very, very wrong. > > IMO-YMMV. No, I do not. But again, where is your "careful test plan" please? All I have seen from you so far is asking us to provide you with a careful test plan while dancing carefully around every single attempt to get a look at what you got so far. I have written PostgreSQL regression tests. I have done consistency checks of entire ERP systems prior to data conversion attempts. I know the value of tests, whether they are against software or data. May I ask what you've done so far? I personally think you don't actually ever did any software testing yourself. You are not really talking from experience, are you? So please, show me what you have now, or get one more plus on my minus-list. Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
> -----Original Message----- > From: Jan Wieck [mailto:JanWieck@Yahoo.com] > Sent: Monday, June 23, 2003 10:30 PM > To: Dann Corbit > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; > PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze [snip] > I personally think you don't actually ever did any software testing > yourself. You are not really talking from experience, are you? So > please, show me what you have now, or get one more plus on my > minus-list. I have already posted enough information to show my qualitications. Whether I am qualified or not is irrelevant to whether the argument has merit or not. I have raised what I consider to be an important issue. Astute members of the list have noticed that I have not volunteered to perform the work. I may or may not produce some efforts towards testing PostgreSQL. Whether I decide to help or not is irrelevant towards the concept of what needs to be done.
Dann, > Astute members of the list have noticed that I have not volunteered to > perform the work. I may or may not produce some efforts towards testing > PostgreSQL. Whether I decide to help or not is irrelevant towards the > concept of what needs to be done. It is not irrelevant. This is an Open Source project, not some Dot-Com where you can float good ideas until you go bankrupt. If there's no possibility of us getting a major 3rd-party certified battery of QA tests donated, why bother putting it on the TODO list? Would it be nice if we had more tests? Yes. In fact, one of the items on my personal todo list is to devise a more versatile performance test than pgbench for testing postgresql parameters, builds, and installations. But it's not getting done by me carping at people on the Hackers list. It'll get done when I spend a long weekend writing Perl. Put up or shut up time, Dann. -- Josh Berkus Aglio Database Solutions San Francisco
> -----Original Message----- > From: Josh Berkus [mailto:josh@agliodbs.com] > Sent: Monday, June 23, 2003 10:50 PM > To: Dann Corbit; Jan Wieck > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; > PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > Dann, > > > Astute members of the list have noticed that I have not > volunteered to > > perform the work. I may or may not produce some efforts towards > > testing PostgreSQL. Whether I decide to help or not is irrelevant > > towards the concept of what needs to be done. > > It is not irrelevant. This is an Open Source project, not > some Dot-Com where > you can float good ideas until you go bankrupt. If there's > no possibility > of us getting a major 3rd-party certified battery of QA tests > donated, why > bother putting it on the TODO list? > > Would it be nice if we had more tests? Yes. In fact, one of > the items on my > personal todo list is to devise a more versatile performance > test than > pgbench for testing postgresql parameters, builds, and > installations. But > it's not getting done by me carping at people on the Hackers > list. It'll get > done when I spend a long weekend writing Perl. > > Put up or shut up time, Dann. It's shut up, then. I'm not willing to commit to this effort.
Dann Corbit wrote: >> -----Original Message----- >> From: Jan Wieck [mailto:JanWieck@Yahoo.com] >> Sent: Monday, June 23, 2003 10:30 PM >> To: Dann Corbit >> Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; >> PostgreSQL-development >> Subject: Re: [HACKERS] Two weeks to feature freeze > [snip] >> I personally think you don't actually ever did any software testing >> yourself. You are not really talking from experience, are you? So >> please, show me what you have now, or get one more plus on my >> minus-list. > > I have already posted enough information to show my qualitications. > > Whether I am qualified or not is irrelevant to whether the argument has > merit or not. > > I have raised what I consider to be an important issue. > > Astute members of the list have noticed that I have not volunteered to > perform the work. I may or may not produce some efforts towards testing > PostgreSQL. Whether I decide to help or not is irrelevant towards the > concept of what needs to be done. Right! Whatever you decide is totally irrelevant towards the concept of what needs to be done. But that wasn't the question and you wheren't in the position or asked for making any decisions. And after this mail I doubt more than before that the input you can provide will lead to any meaningful improvement of PostgreSQL. Then again, you still have the chance to surprise me. I know by now that you're not a programmer and don't know SQL very well. But do you at least have some concept of your own, other than URL's pointing to someone elses work? Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
On Mon, 2003-06-23 at 21:36, The Hermit Hacker wrote:
> On Mon, 23 Jun 2003, Robert Treat wrote:
>
> > > The target-date-based approach we've taken in the last couple of
> > > releases seems much more productive.
> > >
> >
> > productive on a small scale; for sure. productive for large scale
> > features... well, that's why it's being discussed.
>
> 'K, but if we extend the dev cycle in order to get 'foo' in, how is that
> better then having 'foo' continue to be developed thru the release and
> committed in the next cycle?
>
I think it makes it easier to code 'foo' since you're not coding against
(quite as much of) a moving target. I could also help developers stay
focused on particular projects since they are the "hot potato" for a
given release. It could also help people plan better since they would
know that foo is coming in the next release.
> If it takes foo 6 months to develop, I'd rather have the release happen
> after 4 months as per normal (or close to it) and have 'foo' brought in
> part way into dev cycle 2 ... at least then, if 'foo' ends up taking 7
> months, we aren't delaying even further ...
>
i'm sure everyone who doesn't want foo would agree with that position.
The counter though is those folks who did want foo but now have to wait
4-6 months for the next release since foo took a month longer to develop
than was initially planned.
> Its not like our dev cycles have 'idle periods' where nothing is happening
> and we're waiting for a feature to come along ... there is *alot* of
> changes going on that affect alot of ppl that don't really care about
> feature 'foo' coming along ...
>
this doesn't really change anything for those folks, since the only
rational is "every 6 months we do a release." ie. there are *alot* of
changes going on that affect alot of ppl that don't really care about
waiting n more months...
the whole discussion is based on how do we get big projects done when no
one is motivated to work on 'foo' until there faced with a deadline;
this idea puts the pressure on 'foo' developers from the get go. i'm not
saying this a guaranteed way to solve that problem but i think it is a
possible solution. i'm sure there will be big projects most people don't
care about (win32) and big projects most people would like (replication)
so the amount of like or dislike of the idea would be relative in
practice, the key question is would this actually motivate folks more to
get big projects finished faster? since there are only a handful of
folks who fall into that category they can decide for themselves, and if
it wouldn't motivate them, then the question can be asked again, what
would?
Robert Treat
--
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL
Robert Treat wrote: > the whole discussion is based on how do we get big projects done when no > one is motivated to work on 'foo' until there faced with a deadline; > this idea puts the pressure on 'foo' developers from the get go. i'm not > saying this a guaranteed way to solve that problem but i think it is a > possible solution. i'm sure there will be big projects most people don't > care about (win32) and big projects most people would like (replication) > so the amount of like or dislike of the idea would be relative in > practice, the key question is would this actually motivate folks more to > get big projects finished faster? since there are only a handful of > folks who fall into that category they can decide for themselves, and if > it wouldn't motivate them, then the question can be asked again, what > would? I can confirm that there are several people working on Win32/PITR right now, maybe four, that wouldn't if we hadn't set the beta freeze at July 1 --- so such pressure is a real motivator --- though certainly this isn't the way we want to motivate developers. -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
'K, and do you have any ETA on when you'll have this translated into some useful tests that we can incorporate? On Mon, 23 Jun 2003, Dann Corbit wrote: > Here is a list of a small sample of the citations available from the ACM > on software testing: > > http://portal.acm.org/citation.cfm?id=581358&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=376180&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=367020&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=308790&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=257668&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=248262&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > http://portal.acm.org/citation.cfm?id=227759&coll=portal&dl=ACM&CFID=657 > 0092&CFTOKEN=81653602 > > These articles are careful, scientific studies that have been > extensively peer reviewed. > > These articles indicate that testing is a good idea. > > ---------------------------(end of broadcast)--------------------------- > TIP 6: Have you searched our list archives? > > http://archives.postgresql.org > Marc G. Fournier ICQ#7615664 IRC Nick: Scrappy Systems Administrator @ hub.org primary: scrappy@hub.org secondary: scrappy@{freebsd|postgresql}.org
On Mon, 23 Jun 2003, Dann Corbit wrote: > > -----Original Message----- > > From: Jan Wieck [mailto:JanWieck@Yahoo.com] > > Sent: Monday, June 23, 2003 10:30 PM > > To: Dann Corbit > > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; > > PostgreSQL-development > > Subject: Re: [HACKERS] Two weeks to feature freeze > [snip] > > I personally think you don't actually ever did any software testing > > yourself. You are not really talking from experience, are you? So > > please, show me what you have now, or get one more plus on my > > minus-list. > > I have already posted enough information to show my qualitications. > > Whether I am qualified or not is irrelevant to whether the argument has > merit or not. > > I have raised what I consider to be an important issue. You have raised a point that you are not prepared to do anything about though ... nobody disagrees with you about adding more testing, but if you aren't willing to do the work, why should anyone else be? Someone has to spearhead it ... you seem to be passionate about seeing it happen, but don't care enough to do anything about it ...
On Mon, 23 Jun 2003, Dann Corbit wrote: > > Would it be nice if we had more tests? Yes. In fact, one of > > the items on my > > personal todo list is to devise a more versatile performance > > test than > > pgbench for testing postgresql parameters, builds, and > > installations. But > > it's not getting done by me carping at people on the Hackers > > list. It'll get > > done when I spend a long weekend writing Perl. > > > > Put up or shut up time, Dann. > > It's shut up, then. I'm not willing to commit to this effort. Woo hoo ... now *that* was the longest, useless thread I think we've had here so far .. we almost need to start a 'hall of shameful threads' ...
On Tue, 24 Jun 2003, Robert Treat wrote: > On Mon, 2003-06-23 at 21:36, The Hermit Hacker wrote: > > On Mon, 23 Jun 2003, Robert Treat wrote: > > > > > > The target-date-based approach we've taken in the last couple of > > > > releases seems much more productive. > > > > > > > > > > productive on a small scale; for sure. productive for large scale > > > features... well, that's why it's being discussed. > > > > 'K, but if we extend the dev cycle in order to get 'foo' in, how is that > > better then having 'foo' continue to be developed thru the release and > > committed in the next cycle? > > > > I think it makes it easier to code 'foo' since you're not coding against > (quite as much of) a moving target. I *soooooo* disagree with this one ... the only way that postgresql is going to stop being a moving target so that someone can code 'foo' is if everyone else goes to sleep until that happens ... > It could also help people plan better since they would know that foo is > coming in the next release. 'K, that helps the end users, but that doesn't do anything for the person developing 'foo' ... > i'm sure everyone who doesn't want foo would agree with that position. > The counter though is those folks who did want foo but now have to wait > 4-6 months for the next release since foo took a month longer to develop > than was initially planned. Ya, but, based on past experience (hell, this release has been a good example) ... you delay the release because developer of 'foo' figures "oh, it will be ready in another month", and then something comes up that causes that "commitment" not to happen, so we delay it "yet another month"? And I'm not saying that the second delay was due to mis-estimating the time needed ... suddenly getting really busy on a contract, a day job, a death in the family, etc ... you cannot predict what could cause a delay, or how long such a delay would take ... > the whole discussion is based on how do we get big projects done when no > one is motivated to work on 'foo' until there faced with a deadline; > this idea puts the pressure on 'foo' developers from the get go. i'm not > saying this a guaranteed way to solve that problem but i think it is a > possible solution. i'm sure there will be big projects most people don't > care about (win32) and big projects most people would like (replication) > so the amount of like or dislike of the idea would be relative in > practice, the key question is would this actually motivate folks more to > get big projects finished faster? since there are only a handful of > folks who fall into that category they can decide for themselves, and if > it wouldn't motivate them, then the question can be asked again, what > would? First, we already seen that such a 'pressure' doesn't matter, especially if when push comes to shove, they know we'll postpone to accommodate them ... Second ... I don't believe the problem *is* the release cycles ... I think the problem that needs a solution is how to "open up" these large projects so that the deadline(s) don't fall on ones person's shoulders to get done .. how do we encourage/promote "work groups" for the large projects?
> -----Original Message----- > From: The Hermit Hacker [mailto:scrappy@postgresql.org] > Sent: Tuesday, June 24, 2003 5:26 PM > To: Dann Corbit > Cc: Jan Wieck; scott.marlowe; Bruce Momjian; Tom Lane; Jason > Earl; PostgreSQL-development > Subject: Re: [HACKERS] Two weeks to feature freeze > > > On Mon, 23 Jun 2003, Dann Corbit wrote: > > > > -----Original Message----- > > > From: Jan Wieck [mailto:JanWieck@Yahoo.com] > > > Sent: Monday, June 23, 2003 10:30 PM > > > To: Dann Corbit > > > Cc: scott.marlowe; Bruce Momjian; Tom Lane; Jason Earl; > > > PostgreSQL-development > > > Subject: Re: [HACKERS] Two weeks to feature freeze > > [snip] > > > I personally think you don't actually ever did any > software testing > > > yourself. You are not really talking from experience, are you? So > > > please, show me what you have now, or get one more plus on my > > > minus-list. > > > > I have already posted enough information to show my qualitications. > > > > Whether I am qualified or not is irrelevant to whether the argument > > has merit or not. > > > > I have raised what I consider to be an important issue. > > You have raised a point that you are not prepared to do > anything about though I did something about it. I raised the issue. Is it really so that whoever it is that raises a question is also the one who must fix the issue raised? A strange model indeed. > ... nobody disagrees with you about > adding more testing, Actually some do, but that's neither here nor there. > but if you aren't willing to do the > work, why should anyone else be? Perhaps they have the time and care about the result. > Someone has to spearhead it > ... you seem to be passionate about seeing it happen, but > don't care enough to do anything about it ... Don't care and won't do are not the same thing.
On Tue, 24 Jun 2003, Dann Corbit wrote: > I did something about it. I raised the issue. > Is it really so that whoever it is that raises a question is also the > one who must fix the issue raised? > A strange model indeed. Its worked for us ... Wait, I know what should make you happy ... it won't get anytihng done, but ... Bruce, can you add "* Improve Testing" to the TODO list > > Someone has to spearhead it > > ... you seem to be passionate about seeing it happen, but > > don't care enough to do anything about it ... > > Don't care and won't do are not the same thing. Well, actually, they are ... if someone doesn't care, they aren't going to do, are they?
The Hermit Hacker wrote: > On Tue, 24 Jun 2003, Dann Corbit wrote: > > > I did something about it. I raised the issue. > > Is it really so that whoever it is that raises a question is also the > > one who must fix the issue raised? > > A strange model indeed. > > Its worked for us ... > > Wait, I know what should make you happy ... it won't get anytihng done, > but ... > > Bruce, can you add "* Improve Testing" to the TODO list That seems too vague for TODO. We might as well add "Make PostgreSQL faster". :-) -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
> -----Original Message----- > From: The Hermit Hacker [mailto:scrappy@postgresql.org] > Sent: Tuesday, June 24, 2003 6:10 PM > To: Dann Corbit > Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce > Momjian; Tom Lane; Jason Earl; PostgreSQL-development > Subject: RE: [HACKERS] Two weeks to feature freeze > > > On Tue, 24 Jun 2003, Dann Corbit wrote: > > > I did something about it. I raised the issue. > > Is it really so that whoever it is that raises a question > is also the > > one who must fix the issue raised? A strange model indeed. > > Its worked for us ... > > Wait, I know what should make you happy ... it won't get > anytihng done, but ... > > Bruce, can you add "* Improve Testing" to the TODO list It is surely a titanic mistake to bring up any issue on this list if you do not plan to fix it yourself. > > > Someone has to spearhead it > > > ... you seem to be passionate about seeing it happen, but > don't care > > > enough to do anything about it ... > > > > Don't care and won't do are not the same thing. > > Well, actually, they are ... if someone doesn't care, they > aren't going to do, are they? You have had the time to do everything you ever cared about? It is really true that I have made a titanic waste of time in an effort to get someone else to do something about it or at least get them interested in it. I won't waste my time in that way again. I deeply rue the time I have wasted already.
I think it was a useful discussion. I find it interesting to compare our clearly ad-hock testing methods to traditional commercial testing strategies. I think our results are very good, but it does look awful "ad-hock" and it is easy to see how someone would question its effectiveness. Of course, the whole open source development model seems ad-hock too, and on the surface seems inferior to a commercial software development model, but there again, the proof is in the result. --------------------------------------------------------------------------- Dann Corbit wrote: > > -----Original Message----- > > From: The Hermit Hacker [mailto:scrappy@postgresql.org] > > Sent: Tuesday, June 24, 2003 6:10 PM > > To: Dann Corbit > > Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce > > Momjian; Tom Lane; Jason Earl; PostgreSQL-development > > Subject: RE: [HACKERS] Two weeks to feature freeze > > > > > > On Tue, 24 Jun 2003, Dann Corbit wrote: > > > > > I did something about it. I raised the issue. > > > Is it really so that whoever it is that raises a question > > is also the > > > one who must fix the issue raised? A strange model indeed. > > > > Its worked for us ... > > > > Wait, I know what should make you happy ... it won't get > > anytihng done, but ... > > > > Bruce, can you add "* Improve Testing" to the TODO list > > It is surely a titanic mistake to bring up any issue on this list if you > do not plan to fix it yourself. > > > > > Someone has to spearhead it > > > > ... you seem to be passionate about seeing it happen, but > > don't care > > > > enough to do anything about it ... > > > > > > Don't care and won't do are not the same thing. > > > > Well, actually, they are ... if someone doesn't care, they > > aren't going to do, are they? > > You have had the time to do everything you ever cared about? > > It is really true that I have made a titanic waste of time in an effort > to get someone else to do something about it or at least get them > interested in it. I won't waste my time in that way again. I deeply > rue the time I have wasted already. > > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
On Tue, 24 Jun 2003, Bruce Momjian wrote: > The Hermit Hacker wrote: > > On Tue, 24 Jun 2003, Dann Corbit wrote: > > > > > I did something about it. I raised the issue. > > > Is it really so that whoever it is that raises a question is also the > > > one who must fix the issue raised? > > > A strange model indeed. > > > > Its worked for us ... > > > > Wait, I know what should make you happy ... it won't get anytihng done, > > but ... > > > > Bruce, can you add "* Improve Testing" to the TODO list > > That seems too vague for TODO. We might as well add "Make PostgreSQL > faster". :-) 'K, can you add that one too? :)
On Tue, 24 Jun 2003, Dann Corbit wrote: > > > Don't care and won't do are not the same thing. > > > > Well, actually, they are ... if someone doesn't care, they > > aren't going to do, are they? > > You have had the time to do everything you ever cared about? No no, that isn't what he is arguing (or I'm miss reading) ... he said that "not caring about something *and* not doing it aren't the same thing" ... which isnt' the same as caring but not having the time ... is it?
The Hermit Hacker wrote: > On Tue, 24 Jun 2003, Dann Corbit wrote: > >> I did something about it. I raised the issue. >> Is it really so that whoever it is that raises a question is also the >> one who must fix the issue raised? >> A strange model indeed. > > Its worked for us ... Sorry Marc, but that ain't entirely true. There have been a good number of examples where the one who raised an issue isn't just of the format to implement it. So someone else jumped in and did it instead. I don't need to pick any particular samples, you know that it happened a few times. And don't get the wrong picture. Yes, Dann is "just talking" here on the testing methodology front. But as much as it looks like we two hate each other on this thread, we actually start working together on a totally different issue. And to say the least, I like his version better than Katie's ... 'nuff said. Jan > > Wait, I know what should make you happy ... it won't get anytihng done, > but ... > > Bruce, can you add "* Improve Testing" to the TODO list > >> > Someone has to spearhead it >> > ... you seem to be passionate about seeing it happen, but >> > don't care enough to do anything about it ... >> >> Don't care and won't do are not the same thing. > > Well, actually, they are ... if someone doesn't care, they aren't going to > do, are they? -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
The Hermit Hacker wrote:
> On Tue, 24 Jun 2003, Bruce Momjian wrote:
>
>> The Hermit Hacker wrote:
>> > On Tue, 24 Jun 2003, Dann Corbit wrote:
>> >
>> > > I did something about it. I raised the issue.
>> > > Is it really so that whoever it is that raises a question is also the
>> > > one who must fix the issue raised?
>> > > A strange model indeed.
>> >
>> > Its worked for us ...
>> >
>> > Wait, I know what should make you happy ... it won't get anytihng done,
>> > but ...
>> >
>> > Bruce, can you add "* Improve Testing" to the TODO list
>>
>> That seems too vague for TODO. We might as well add "Make PostgreSQL
>> faster". :-)
>
> 'K, can you add that one too? :)
Make his life easier:
Replace the entire TODO with "Make PostgreSQL better"
That pretty much summs it up, no?
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #
On Tue, 24 Jun 2003, Jan Wieck wrote: > The Hermit Hacker wrote: > > On Tue, 24 Jun 2003, Bruce Momjian wrote: > > > >> The Hermit Hacker wrote: > >> > On Tue, 24 Jun 2003, Dann Corbit wrote: > >> > > >> > > I did something about it. I raised the issue. > >> > > Is it really so that whoever it is that raises a question is also the > >> > > one who must fix the issue raised? > >> > > A strange model indeed. > >> > > >> > Its worked for us ... > >> > > >> > Wait, I know what should make you happy ... it won't get anytihng done, > >> > but ... > >> > > >> > Bruce, can you add "* Improve Testing" to the TODO list > >> > >> That seems too vague for TODO. We might as well add "Make PostgreSQL > >> faster". :-) > > > > 'K, can you add that one too? :) > > Make his life easier: > > Replace the entire TODO with "Make PostgreSQL better" > > That pretty much summs it up, no? Oh, I like that ... definitely leaves it open to the interpretation of the reader as to what would make it better :)
On Tue, 24 Jun 2003, Dann Corbit wrote: > > -----Original Message----- > > From: The Hermit Hacker [mailto:scrappy@postgresql.org] > > Sent: Tuesday, June 24, 2003 6:10 PM > > To: Dann Corbit > > Cc: The Hermit Hacker; Jan Wieck; scott.marlowe; Bruce > > Momjian; Tom Lane; Jason Earl; PostgreSQL-development > > Subject: RE: [HACKERS] Two weeks to feature freeze > > > > > > On Tue, 24 Jun 2003, Dann Corbit wrote: > > > > > I did something about it. I raised the issue. > > > Is it really so that whoever it is that raises a question > > is also the > > > one who must fix the issue raised? A strange model indeed. > > > > Its worked for us ... > > > > Wait, I know what should make you happy ... it won't get > > anytihng done, but ... > > > > Bruce, can you add "* Improve Testing" to the TODO list > > It is surely a titanic mistake to bring up any issue on this list if you > do not plan to fix it yourself. No, it's not. But you have to realize that when you bring up a "problem" without a solution, you are, in effect, asking someone else to solve your problem. I do that all the time. I don't code postgresql (well, I've hacked around in it a bit for fun, but most of it is over my head.) So, then I bring up something (like the screwy date mangling / parsing misfeature we've been discussing this last week) I KNOW I'm asking the programmers for a favor to fix it, and I know that if they all said, no, we don't have time to fix it, if you want it done you'd better get hacking, then I can either get hacking or wait until someone has time to fix it. I.e. I have to be civil and present my case in a convincing enough manner to get someone else to do it. The bigger the change, the more likely it is that it'll get thrown back at the person suggesting it. A prime example is using a threaded model. About every three months or so someone shows up waving their hands about how Postgresql just can't be fast without threads. No code, no benchmarks, no proof, just a rough feeling they got from reading a magazine article. Invariably, the issue is far more complex than just "make Postgresql threaded" and the person suggesting it can "justify" why someone else should spend months doing it, but they can't be bothered themselves. In the Open Source universe, if you want a feature, you have to be willing to "pay for it." Whether that's with code or a checkbook, either way, TANSTAAFL. In the case of better testing, if you're willing to get out your checkbook and start pumping out cash to a few of the developers here on the list, I'd sure someone would jump right up and start writing your test suite. Or, you can code it yourself. Or, you can convince someone that the testing is necessary and they'll volunteer their time. For me, more testing would gain me nothing. I already test postgresql on a test box before throwing it into production. I test it for load, response, correctness, etc... and if I miss something, it's usually a pretty small issue, and it gets fixed fast. I'd much rather see time and effort go into the things they go into, like optimizing the query planner, in() with subselects, full text indexing, and many other things. There is a finite pool of hacker skill poured into Postgresql, and pouring more of it into testing means less gets poured into development, and right now, I've got all the testing I need, as do most of the developers and users I know. Open Source works because some scratches and itch. right now, you're the only one with an itch for more testing. If you don't scratch it, it likely won't get scratched any time soon. who knows, in a year, maybe someone else will get the itch and scratch it. But your arguments have been unconvincing, since the reality of Postgresql is that it is the absolutely most reliable database I know of, and has one of the fastest turnaround times for bug fixes when they do pop up. Does apache have a comprehensive test suite? Not that I know of. Neither does the Linux kernel or the BSD kernel. But, they all get the job done better than their commercial counterparts with far less headache. This discussion hasn't been a complete waste of time, but your arguments have bordered on insulting to those of us who currently DO the testing on our own machines. You've dismissed our testing out of hand on several occasions as insufficient, when at least we ARE testing postgresql, even if it doesn't meat up to your higher standards. If you want to raise the bar, then YOU get to raise it.
> > That seems too vague for TODO. We might as well add "Make PostgreSQL > > faster". :-) > 'K, can you add that one too? :) How about "* Remove all bugs from the source". Can you put that in TODO ? :-) -- Kaare Rasmussen --Linux, spil,-- Tlf: 3816 2582 Kaki Data tshirts, merchandize Fax: 3816 2501 Howitzvej 75 Åben 12.00-18.00 Email: kar@kakidata.dk 2000 Frederiksberg Lørdag 12.00-16.00 Web: www.suse.dk
Kaare Rasmussen wrote: >> > That seems too vague for TODO. We might as well add "Make PostgreSQL >> > faster". :-) >> 'K, can you add that one too? :) > > How about "* Remove all bugs from the source". Can you put that in TODO ? > > :-) > Change that into "* Remove bugs from source code" and get a patent on it. Should be a nobrainer (as in those guy's have no brains) considering that NetFlix even got a patent on DVD subscription rentals: http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99 Jan -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Jan Wieck wrote: > > Change that into "* Remove bugs from source code" and get a patent on > it. Should be a nobrainer (as in those guy's have no brains) > considering that NetFlix even got a patent on DVD subscription rentals: > > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99 I'm applying for a patent on breathing now. The trick I found is reversing the direction of airflow in a regular way. The algorithm seems apparently simple, but it really makes the deal: Step 1. If your lungs are empty, let air flow into them through some air intake. This airflow might be ducted by some bronchial or additional tubing. Step 2 (optional) As soon as there's enough air in the lungs, you may decide to hold it there for a while. Some time limits might apply, please consult some specialist for details. Step 3 Press the air out of the lungs, using some muscles or externally applied force on the chest. The air will eventually escape through some body opening. Another patent I'll be applying for covers the use of nostrils for this purpose. Step 4 Restart the cycle at step 1. Regards and don't dare to try this without royalty fees! Andreas
On 25 Jun 2003 at 14:50, Andreas Pflug wrote: > Jan Wieck wrote: > > Change that into "* Remove bugs from source code" and get a patent on > > it. Should be a nobrainer (as in those guy's have no brains) > > considering that NetFlix even got a patent on DVD subscription rentals: > > > > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99 > > I'm applying for a patent on breathing now. > The trick I found is reversing the direction of airflow in a regular way. > > The algorithm seems apparently simple, but it really makes the deal: > > Step 1. > If your lungs are empty, let air flow into them through some air intake. > This airflow might be ducted by some bronchial or additional tubing. > > Step 2 (optional) > As soon as there's enough air in the lungs, you may decide to hold it > there for a while. Some time limits might apply, please consult some > specialist for details. > > Step 3 > Press the air out of the lungs, using some muscles or externally applied > force on the chest. The air will eventually escape through some body > opening. Another patent I'll be applying for covers the use of nostrils > for this purpose. > > Step 4 > Restart the cycle at step 1. > > > Regards and don't dare to try this without royalty fees! Oops... I challenge it for prior art..:-) ByeShridhar -- Gomme's Laws: (1) A backscratcher will always find new itches. (2) Time accelerates. (3) The weather at home improves as soon as you go away.
> Change that into "* Remove bugs from source code" and get a patent on > it. Should be a nobrainer (as in those guy's have no brains) considering > that NetFlix even got a patent on DVD subscription rentals: And for all the nice royalty money*, think about what can be done to PostgreSQL. Maybe even a test suite :-) * Or maybe the best step is to sue IBM or Microsoft. Then again, maybe not. Do they care about bug removal? ;-) -- Kaare Rasmussen --Linux, spil,-- Tlf: 3816 2582 Kaki Data tshirts, merchandize Fax: 3816 2501 Howitzvej 75 Åben 12.00-18.00 Email: kar@kakidata.dk 2000 Frederiksberg Lørdag 12.00-16.00 Web: www.suse.dk
Jan, > There have been a good number of examples where the one who raised an > issue isn't just of the format to implement it. So someone else jumped > in and did it instead. I don't need to pick any particular samples, you > know that it happened a few times. Sure. But in those cases, the fix/feature was something that the consensus on -Hackers agreed needed to be done, someday. Putting those items on the TODO was an acknowledgement of that decision, even though they won't be implemented until we aquire more contributors. As an example, the various PL/pgSQL enhancements which Patrick and I pushed onto the TODO list, which are still features in search of a programmer. That took, as I recall, about a month of lobbying on this list, which required me to prove a) how our current PL/pgSQL was weak, and b) how the improvements would benefit the project overall, and c) how many people were interested in the improvements. There are 3 problems with Dann's proposal that have caused it to be shot down in flames instead of being put on the TODO list: 1) Most people on this list ... particularly, most contributors ... do not agree with Dann's proposal. This is in no little part due to Dann's lack of material evidence for his case. 2) Dann has a particularly abrasive and insulting communication style that hasn't helped his case any, either. 3) Dann is proposing not just a feature but sweeping changes to the way our commmunity works, despite having been a member of this community for about 3 weeks total. As such, this discussion is an example of the "Open Source Process" (tm) working correctly. Someone brought up a proposal, it was challenged, they were not able to defend the proposal, and it was voted down. -- Josh Berkus Aglio Database Solutions San Francisco
Josh Berkus <josh@agliodbs.com> writes:
> 3) Dann is proposing not just a feature but sweeping changes to the way our
> commmunity works, despite having been a member of this community for about 3
> weeks total.
In Dann's defense, I didn't think I heard him proposing that we get rid
of our existing testing methods, but rather that we see if we can't
supplement them with something more formal. This strikes me as a
perfectly reasonable proposal. However, he hasn't succeeded in
convincing anyone else to put their time into it (for reasons that
are also perfectly reasonable, namely that we find that our existing
methods do pretty well, and we don't have the manpower to create a large
formal testing structure ... even if we thought it would repay the effort,
which many of us doubt). So it's his itch to scratch, or not.
Unless there's something more profitable to be said on the subject,
could we drop the thread?
regards, tom lane
Tom Lane wrote: > Josh Berkus <josh@agliodbs.com> writes: > > 3) Dann is proposing not just a feature but sweeping changes to the way our > > commmunity works, despite having been a member of this community for about 3 > > weeks total. > > In Dann's defense, I didn't think I heard him proposing that we get rid > of our existing testing methods, but rather that we see if we can't > supplement them with something more formal. This strikes me as a > perfectly reasonable proposal. However, he hasn't succeeded in > convincing anyone else to put their time into it (for reasons that > are also perfectly reasonable, namely that we find that our existing > methods do pretty well, and we don't have the manpower to create a large > formal testing structure ... even if we thought it would repay the effort, > which many of us doubt). So it's his itch to scratch, or not. > > Unless there's something more profitable to be said on the subject, > could we drop the thread? One thing that came out of the thread is the fact that many people who use PostgreSQL do testing in many different ways, and that much of the stability of PostgreSQL can be attributed to that. It occurs to me that there may be (perhaps) a lot of duplication of effort there that could be reduced a bit. So...would it make sense to create a gborg project to which people who have written their own test suites can contribute whatever code and data they feel comfortable releasing? As a gborg project, it would be separate from the main PG distribution and would thus have no impact on the build process or anything like that. But at the same time, if there are any ideas on testing that people have had, they could be shared with others through that mechanism. And any tests which prove to be particularly useful could make their way into the PG distribution if people here wish. Of course, like anything else this could be a bad (or perhaps redundant) idea. :-) -- Kevin Brown kevin@sysexperts.com
Kevin, > So...would it make sense to create a gborg project to which people who > have written their own test suites can contribute whatever code and > data they feel comfortable releasing? As a gborg project, it would be > separate from the main PG distribution and would thus have no impact on > the build process or anything like that. But at the same time, if there > are any ideas on testing that people have had, they could be shared with > others through that mechanism. +1 -Josh
On Wed, 25 Jun 2003, Shridhar Daithankar wrote: > On 25 Jun 2003 at 14:50, Andreas Pflug wrote: > > Jan Wieck wrote: > > > Change that into "* Remove bugs from source code" and get a patent on > > > it. Should be a nobrainer (as in those guy's have no brains) > > > considering that NetFlix even got a patent on DVD subscription rentals: > > > > > > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99 > > > > I'm applying for a patent on breathing now. > > The trick I found is reversing the direction of airflow in a regular way. > > > > The algorithm seems apparently simple, but it really makes the deal: > > > > Step 1. > > If your lungs are empty, let air flow into them through some air intake. > > This airflow might be ducted by some bronchial or additional tubing. > > > > Step 2 (optional) > > As soon as there's enough air in the lungs, you may decide to hold it > > there for a while. Some time limits might apply, please consult some > > specialist for details. > > > > Step 3 > > Press the air out of the lungs, using some muscles or externally applied > > force on the chest. The air will eventually escape through some body > > opening. Another patent I'll be applying for covers the use of nostrils > > for this purpose. > > > > Step 4 > > Restart the cycle at step 1. > > > > > > Regards and don't dare to try this without royalty fees! > > Oops... I challenge it for prior art..:-) Why, are you older then he is? I think that is about the only way that 'prior art' would apply here, no? :)
On Wed, 25 Jun 2003, Kevin Brown wrote: > So...would it make sense to create a gborg project to which people who > have written their own test suites can contribute whatever code and data > they feel comfortable releasing? As a gborg project, it would be > separate from the main PG distribution and would thus have no impact on > the build process or anything like that. But at the same time, if there > are any ideas on testing that people have had, they could be shared with > others through that mechanism. > > And any tests which prove to be particularly useful could make their way > into the PG distribution if people here wish. > > Of course, like anything else this could be a bad (or perhaps redundant) > idea. :-) It doesn't sound like a bad idea ... but, it pretty much comes down to the original thread: are you willing to step up and maintain such a project?
Doesn't matter, this entire approach has a fundamental flaw. If the lungs are "empty" ... that means that the guy has an open thorax, the lungs are collapsed, and you'll probably have problems catching his attention to make your claim. Jan The Hermit Hacker wrote: > On Wed, 25 Jun 2003, Shridhar Daithankar wrote: > >> On 25 Jun 2003 at 14:50, Andreas Pflug wrote: >> > Jan Wieck wrote: >> > > Change that into "* Remove bugs from source code" and get a patent on >> > > it. Should be a nobrainer (as in those guy's have no brains) >> > > considering that NetFlix even got a patent on DVD subscription rentals: >> > > >> > > http://slashdot.org/article.pl?sid=03/06/24/1458223&mode=flat&tid=155&tid=99 >> > >> > I'm applying for a patent on breathing now. >> > The trick I found is reversing the direction of airflow in a regular way. >> > >> > The algorithm seems apparently simple, but it really makes the deal: >> > >> > Step 1. >> > If your lungs are empty, let air flow into them through some air intake. >> > This airflow might be ducted by some bronchial or additional tubing. >> > >> > Step 2 (optional) >> > As soon as there's enough air in the lungs, you may decide to hold it >> > there for a while. Some time limits might apply, please consult some >> > specialist for details. >> > >> > Step 3 >> > Press the air out of the lungs, using some muscles or externally applied >> > force on the chest. The air will eventually escape through some body >> > opening. Another patent I'll be applying for covers the use of nostrils >> > for this purpose. >> > >> > Step 4 >> > Restart the cycle at step 1. >> > >> > >> > Regards and don't dare to try this without royalty fees! >> >> Oops... I challenge it for prior art..:-) > > Why, are you older then he is? I think that is about the only way that > 'prior art' would apply here, no? :) > > > ---------------------------(end of broadcast)--------------------------- > TIP 5: Have you checked our extensive FAQ? > > http://www.postgresql.org/docs/faqs/FAQ.html -- #======================================================================# # It's easier to get forgiveness for being wrong than for being right. # # Let's break this rule - forgive me. # #================================================== JanWieck@Yahoo.com #
Tom Lane wrote: >Peter Eisentraut <peter_e@gmx.net> writes: > > >>Thomas Swan writes: >> >> >>>Have you considered something similar to the Mozilla tinderbox approach >>>where you have a daemon checkout the cvs, compile, run regression tests, >>>and report a status or be able to report a status? >>> >>> > > > >>Even if you could achieve near complete coverage of the platforms, >>platform versions, and auxilliary software versions and combinations that >>PostgreSQL runs with, in most cases, something breaks on a new >>version or combination of these things. >> >> > >Still, whenever we're doing something that interacts at all with the OS, >it seems we get breakages that don't show in the original author's >testing, but only pop up days to months later when some beta tester >tries the code on platform P or using option Q. The current >difficulties with the IPv6 patches are a fine case in point. >If we could get feedback more easily about whether a proposed patch >compiles and passes regression on a variety of platforms, we could >reduce the pain involved by a great deal, simply because the problems >could be fixed while the code is still fresh in mind. > >I don't think there is any company involved with Postgres that is >willing to commit the resources to run a Mozilla-style tinderbox setup >singlehanded. But I wonder whether we couldn't set up something that is >community-based: get a few dozen people with different platforms to >volunteer to check the code regularly on their own machines. I'm >imagining a cron job that fires daily in the wee hours, pulls the latest >CVS tip, does "make distclean; configure; make; make check", and mails >the results to someplace that puts 'em up on our website. > >It's possible that we could adapt the tinderbox software to work this >way, but even if we had to write our own, it seems like a fairly simple >task. And it'd give *much* better feedback on porting problems than we >have now. Sure, there will always be corner cases you don't catch, >but the first rule of testing is the sooner you find a bug the cheaper >it is to fix. > > Is it possible the sourceforge compile farms could be used for some of the automated testing? I'm not sure how that system works, but it could be worth looking into. > regards, tom lane > >---------------------------(end of broadcast)--------------------------- >TIP 2: you can get off all lists at once with the unregister command > (send "unregister YourEmailAddressHere" to majordomo@postgresql.org) > >
Thomas Swan <tswan@idigx.com> writes:
> Is it possible the sourceforge compile farms could be used for some of
> the automated testing? I'm not sure how that system works, but it could
> be worth looking into.
The last time I used it (which admittedly was a year or two back), they
didn't really have a very wide range of platforms available. And I
don't think they were offering cron access on the farm machines, so this
would be much more painful to automate than work done locally on
contributors' machines.
regards, tom lane
On Thu, 26 Jun 2003, Thomas Swan wrote: > > > Is it possible the sourceforge compile farms could be used for some of > the automated testing? I'm not sure how that system works, but it could > be worth looking into. Isn't the sourceforge license very scary and along the lines of "whatever you put on here we own it's just we tend not to persue that at the moment as there's not much money in it for us but that doesn't stop us from claiming it at some indeterminate time in the future"? -- Nigel J. Andrews
Nigel J. Andrews wrote: >On Thu, 26 Jun 2003, Thomas Swan wrote: > > >>Is it possible the sourceforge compile farms could be used for some of >>the automated testing? I'm not sure how that system works, but it could >>be worth looking into. >> >> > >Isn't the sourceforge license very scary and along the lines of "whatever you >put on here we own it's just we tend not to persue that at the moment as >there's not much money in it for us but that doesn't stop us from claiming it >at some indeterminate time in the future"? > If it's that intrusive, then it was a bad idea. But, I didn't find anything like that on their Terms of Use <http://sourceforge.net/docman/display_doc.php?docid=6048&group_id=1> page. The compiler farm has a relatively small number of platforms, but perhaps it would be enough to get started with at least verifying an automated test would work. See Guide to the Sourceforge Compile Farm <http://sourceforge.net/docman/display_doc.php?docid=762&group_id=1>. In terms of implementation, I was thinking of something like the following. * clean the source, destination directories * pull latest CVS tip down. * record environment / installed packages * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. ) o make clean o configurewith sets of options o compile + log messages + analyze errors ( perhaps gatherstatitistics: warnings, failures, notices, etc.) o (run / install) if successful orun tests + output results (perhaps to HTML) + compare results with expected +record differences if any | gather aggregate information o uninstall / clean up * end loop Perhaps there could be an occasion where the test would be able to put in a corrupt WAL or a corrupt table to do regression tests for recovery of errors. Of course, these are just ideas and I'm not sure how practical it is to do any of them. I just am really concerned about the uninstall/clean up phase and how that can be done in an orderly fashion. Unless the process can start from a clean state again, then it won't be valid. At one point I had even given thought, vainly, to purchasing VMWare for such an occasion. Suggestions?
I know I'm new to this list, but is OSDL's testing capabilities out of the question? On Thu, 2003-06-26 at 13:48, Thomas Swan wrote: > Nigel J. Andrews wrote: > > >On Thu, 26 Jun 2003, Thomas Swan wrote: > > > > > >>Is it possible the sourceforge compile farms could be used for some of > >>the automated testing? I'm not sure how that system works, but it could > >>be worth looking into. > >> > >> > > > >Isn't the sourceforge license very scary and along the lines of "whatever you > >put on here we own it's just we tend not to persue that at the moment as > >there's not much money in it for us but that doesn't stop us from claiming it > >at some indeterminate time in the future"? > > > If it's that intrusive, then it was a bad idea. But, I didn't find > anything like that on their Terms of Use > <http://sourceforge.net/docman/display_doc.php?docid=6048&group_id=1> > page. The compiler farm has a relatively small number of platforms, but > perhaps it would be enough to get started with at least verifying an > automated test would work. See Guide to the Sourceforge Compile Farm > <http://sourceforge.net/docman/display_doc.php?docid=762&group_id=1>. ... -- Austin Gonyou <austin@coremetrics.com> Coremetrics, Inc.
On Thu, 2003-06-26 at 15:00, Austin Gonyou wrote: > I know I'm new to this list, but is OSDL's testing capabilities out of > the question? From what I've seen, OSDL is only concerned with a very very small set of platforms (Linux in a couple of configurations). -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
DOH!. Yes....You're right I totally forgot about that. My apologies. I believe though, that there is a HP testing lab that is somewhat like OSDL, in that they offer OSS free services and many platforms to run on. (used to be compaq's developer testdrive sort of program) I believe it still exists. -----Original Message----- From: Rod Taylor [mailto:rbt@rbt.ca] Sent: Thursday, June 26, 2003 3:06 PM To: Austin Gonyou Cc: Thomas Swan; Nigel J. Andrews; Tom Lane; PostgreSQL Development Subject: Re: Two weeks to feature freeze On Thu, 2003-06-26 at 15:00, Austin Gonyou wrote: > I know I'm new to this list, but is OSDL's testing capabilities out of > the question? From what I've seen, OSDL is only concerned with a very very small set of platforms (Linux in a couple of configurations). -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
> * clean the source, destination directories > * pull latest CVS tip down. Why tip? Lets simply update the current source tree to the most current of whatever branch they had checked out initially. Running it on older stable branches is just as useful. > * record environment / installed packages Virtually impossible, especially if people take to installing some items by hand (we want to test them as well). Recording the configure output on the other hand would be quite useful. > * loop - on different options ( w/ or w/o krb5, w/ or w/o ssl, etc. ) > o make clean > o configure with sets of options > o compile > + log messages > + analyze errors ( perhaps gather statitistics: > warnings, failures, notices, etc.) Any warnings, failures, etc. aside from what is in the 'known' file should be reported -- makes sense. > o (run / install) if successful > o run tests > + output results (perhaps to HTML) > + compare results with expected > + record differences if any | gather aggregate information Standard regression tests should suffice. Any non-ignored errors reported. > o uninstall / clean up Skip this. Cleanup prior to the run. If something is wrong, we may need additional information from the environment. > * end loop > > Perhaps there could be an occasion where the test would be able to put > in a corrupt WAL or a corrupt table to do regression tests for recovery > of errors. These would be interesting extensions to the standard regression suite -- but perhaps require a flag > Of course, these are just ideas and I'm not sure how practical it is to > do any of them. I just am really concerned about the uninstall/clean up > phase and how that can be done in an orderly fashion. Unless the > process can start from a clean state again, then it won't be valid. At I've not tried, but if PostgreSQL can do an 'out of tree' compile it could make it much easier. That said, poor cleanup would be a result of a broken make clean. > one point I had even given thought, vainly, to purchasing VMWare for > such an occasion. Suggestions? Skip VMWare. Find a few volunteers to cron the script (I'll volunteer). I think we should replace Bruce's pgtest script with this one -- with an argument to accept the email address to report to for FAILING cases. Success isn't very interesting if it runs regularly. -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
On Thu, 2003-06-26 at 16:09, Gonyou, Austin wrote: > DOH!. Yes....You're right I totally forgot about that. My apologies. I > believe though, that there is a HP testing lab that is somewhat like OSDL, > in that they offer OSS free services and many platforms to run on. (used to > be compaq's developer testdrive sort of program) I believe it still exists. I've been on Compaq's testdrive boxes before -- very nice. But I'm not sure we want to use those for a testing platform. More to the point, if OSDL benchmarks keep coming along we should have a variety of PostgreSQL benchmarks to choose from. Adding a regression test for performance would be a wise thing to do, but will require dedicated hardware for good numbers (especially if run nightly). Thomas had a good start in mind. Hopefully we can extend it a little once it has been started. -- Rod Taylor <rbt@rbt.ca> PGP Key: http://www.rbt.ca/rbtpub.asc
-----Original Message----- From: Rod Taylor [mailto:rbt@rbt.ca] Sent: Thursday, June 26, 2003 3:33 PM To: Gonyou, Austin Cc: Thomas Swan; Nigel J. Andrews; Tom Lane; PostgreSQL Development Subject: RE: Two weeks to feature freeze On Thu, 2003-06-26 at 16:09, Gonyou, Austin wrote: > DOH!. Yes....You're right I totally forgot about that. My apologies. I > believe though, that there is a HP testing lab that is somewhat like OSDL, > in that they offer OSS free services and many platforms to run on. (used to > be compaq's developer testdrive sort of program) I believe it still exists. >>I've been on Compaq's testdrive boxes before -- very nice. But I'm not >>sure we want to use those for a testing platform. Agreed, but the point was, there is a developer side where developers can test their code for a longer period of time in a more lab-like environment, much the same way as OSDL schedules their testing, etc, but you can try stuff on Alpha and x86, run windows, solaris, Tru64, etc. >>More to the point, if OSDL benchmarks keep coming along we should have a >>variety of PostgreSQL benchmarks to choose from. Adding a regression >>test for performance would be a wise thing to do, but will require >>dedicated hardware for good numbers (especially if run nightly). Understood and I concurr, I just didn't think about the testing scope of OSDL. >>Thomas had a good start in mind. Hopefully we can extend it a little >>once it has been started. I agree, just trying to offer something I thought was more valuable than it may have been. :) >>-- >>Rod Taylor <rbt@rbt.ca> >> >>PGP Key: http://www.rbt.ca/rbtpub.asc
On Thu, 26 Jun 2003, Thomas Swan wrote: > Of course, these are just ideas and I'm not sure how practical it is to > do any of them. I just am really concerned about the uninstall/clean up > phase and how that can be done in an orderly fashion. Unless the > process can start from a clean state again, then it won't be valid. At > one point I had even given thought, vainly, to purchasing VMWare for > such an occasion. Suggestions? Personally ... if you could build up the test script, I think there are enough ppl with more platforms on these lists that would be willing ot run it ... the problem isn't getting the "farm" together, its coming up with the automated (or even semi-automated) tests :(
On Thu, 26 Jun 2003, Rod Taylor wrote: > I think we should replace Bruce's pgtest script with this one -- with an > argument to accept the email address to report to for FAILING cases. > Success isn't very interesting if it runs regularly. that was why I suggested getting it into the tree ... to at least give a starting point to work from ... and I have at least one machine right now that I can run such tests on ... Dual PII with FreeBSD 5.x-CURRENT on her ...
The Hermit Hacker wrote: >On Thu, 26 Jun 2003, Thomas Swan wrote: > > > >>Of course, these are just ideas and I'm not sure how practical it is to >>do any of them. I just am really concerned about the uninstall/clean up >>phase and how that can be done in an orderly fashion. Unless the >>process can start from a clean state again, then it won't be valid. At >>one point I had even given thought, vainly, to purchasing VMWare for >>such an occasion. Suggestions? >> >> > >Personally ... if you could build up the test script, I think there are >enough ppl with more platforms on these lists that would be willing ot run >it ... the problem isn't getting the "farm" together, its coming up with >the automated (or even semi-automated) tests :( > I'll see what I can do... my shell script skills are pretty good, but I'm not sure how to handle the noting changes in the gcc output. My best guess is to just do it a couple of times and force something to change (make an intentional mistake) and see if it can catch it, or at least what changes.
The Hermit Hacker wrote:
> On Wed, 25 Jun 2003, Kevin Brown wrote:
>
> > So...would it make sense to create a gborg project to which people who
> > have written their own test suites can contribute whatever code and data
> > they feel comfortable releasing? As a gborg project, it would be
> > separate from the main PG distribution and would thus have no impact on
> > the build process or anything like that. But at the same time, if there
> > are any ideas on testing that people have had, they could be shared with
> > others through that mechanism.
> >
> > And any tests which prove to be particularly useful could make their way
> > into the PG distribution if people here wish.
> >
> > Of course, like anything else this could be a bad (or perhaps redundant)
> > idea. :-)
>
> It doesn't sound like a bad idea ... but, it pretty much comes down to the
> original thread: are you willing to step up and maintain such a project?
Yes, I am ("how hard can it be?", he asks himself, knowing all the
while that it's a really bad idea to be asking that question. :-).
But I haven't the faintest idea of how or where to even start, so
pointers would be appreciated.
--
Kevin Brown kevin@sysexperts.com
On Thu, 26 Jun 2003, Kevin Brown wrote:
> > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > original thread: are you willing to step up and maintain such a project?
>
> Yes, I am ("how hard can it be?", he asks himself, knowing all the
> while that it's a really bad idea to be asking that question. :-).
> But I haven't the faintest idea of how or where to even start, so
> pointers would be appreciated.
Which, of course, is always the fun part ...
I believe Thomas is going to be starting to work on test scripts, so you
might want to co-ordinate with him ...
On Thu, 26 Jun 2003, Kevin Brown wrote:
> The Hermit Hacker wrote:
> > On Wed, 25 Jun 2003, Kevin Brown wrote:
> >
> > > So...would it make sense to create a gborg project to which people who
> > > have written their own test suites can contribute whatever code and data
> > > they feel comfortable releasing? As a gborg project, it would be
> > > separate from the main PG distribution and would thus have no impact on
> > > the build process or anything like that. But at the same time, if there
> > > are any ideas on testing that people have had, they could be shared with
> > > others through that mechanism.
> > >
> > > And any tests which prove to be particularly useful could make their way
> > > into the PG distribution if people here wish.
> > >
> > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > idea. :-)
> >
> > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > original thread: are you willing to step up and maintain such a project?
>
> Yes, I am ("how hard can it be?", he asks himself, knowing all the
> while that it's a really bad idea to be asking that question. :-).
> But I haven't the faintest idea of how or where to even start, so
> pointers would be appreciated.
Create/modify a script to automate some kind of download/sync, test and
send failure results somewhere. Make it extensible, so that other tests
can be easily added -- preferable in a self contained way. It should grow
from there.
Gavin
See my recent commit of src/tools/pgtest. It might be a good start.
---------------------------------------------------------------------------
Gavin Sherry wrote:
> On Thu, 26 Jun 2003, Kevin Brown wrote:
>
> > The Hermit Hacker wrote:
> > > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > >
> > > > So...would it make sense to create a gborg project to which people who
> > > > have written their own test suites can contribute whatever code and data
> > > > they feel comfortable releasing? As a gborg project, it would be
> > > > separate from the main PG distribution and would thus have no impact on
> > > > the build process or anything like that. But at the same time, if there
> > > > are any ideas on testing that people have had, they could be shared with
> > > > others through that mechanism.
> > > >
> > > > And any tests which prove to be particularly useful could make their way
> > > > into the PG distribution if people here wish.
> > > >
> > > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > > idea. :-)
> > >
> > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > > original thread: are you willing to step up and maintain such a project?
> >
> > Yes, I am ("how hard can it be?", he asks himself, knowing all the
> > while that it's a really bad idea to be asking that question. :-).
> > But I haven't the faintest idea of how or where to even start, so
> > pointers would be appreciated.
>
> Create/modify a script to automate some kind of download/sync, test and
> send failure results somewhere. Make it extensible, so that other tests
> can be easily added -- preferable in a self contained way. It should grow
> from there.
>
> Gavin
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html
>
-- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610)
359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square,
Pennsylvania19073
Rod Taylor <rbt@rbt.ca> writes:
> I've not tried, but if PostgreSQL can do an 'out of tree' compile it
> could make it much easier.
Yes it can, following the usual procedure for autoconfiscated trees:
just invoke configure from an empty directory, egmkdir buildcd build../pgsql/configure
This is something that breaks regularly because few of the key
developers use it :-(. If there were automatic tests that used that
build setup, it would be a good thing 'cause it'd keep us honest.
> That said, poor cleanup would be a result of
> a broken make clean.
Not to worry, the developers will notice that case fast enough.
I think the auto test script should just rm -rf build and then
proceed as above.
regards, tom lane
On Thu, 26 Jun 2003, Bruce Momjian wrote:
>
> See my recent commit of src/tools/pgtest. It might be a good start.
Yes this is a good start. This is a little concerning though:
pg_ctl stop
rm -rf "$PGDATA"
Perhaps a warning is warranted (ie, $PGDATA shouldn't point to your
production data cluster)?
On another point, I have some ideas for Kevin and others interested in
automated testing. Dann, Tom and others have voiced concern about the
nature of testing itself: progammers testing for bugs they've solved; the
difficulty/impossibility of testing for conditions you are unaware of,
etc.
ISTM that most of the bugs which aren't caught by the programmer, peer
review and regresion test are revealed by users throwing data into a new
version or a version different to that they are running in production and
then running their existing code against it. That is, bugs are revealed by
usage which developers did not foresee or did not think to test.
So, if we had an automated testing framework which was extensible,
postgres users could create testing scripts which simultate their
application, with their application data (real or created specifically
for the test). The win for users is that they can have their data/SQL
tested on a variety of platforms, on new versions of postgres and the win
for developers/testers is exposure of the code to unexpected usage.
There will need to be checks and balances involved (select 1; is a pretty
ordinary test), size limits/configurable thresholds for run times, and a
repository of test results.
Naturally, managing this could be quite time consuming if data formats
change etc. but if people are keen...
Gavin
>
> ---------------------------------------------------------------------------
>
> Gavin Sherry wrote:
> > On Thu, 26 Jun 2003, Kevin Brown wrote:
> >
> > > The Hermit Hacker wrote:
> > > > On Wed, 25 Jun 2003, Kevin Brown wrote:
> > > >
> > > > > So...would it make sense to create a gborg project to which people who
> > > > > have written their own test suites can contribute whatever code and data
> > > > > they feel comfortable releasing? As a gborg project, it would be
> > > > > separate from the main PG distribution and would thus have no impact on
> > > > > the build process or anything like that. But at the same time, if there
> > > > > are any ideas on testing that people have had, they could be shared with
> > > > > others through that mechanism.
> > > > >
> > > > > And any tests which prove to be particularly useful could make their way
> > > > > into the PG distribution if people here wish.
> > > > >
> > > > > Of course, like anything else this could be a bad (or perhaps redundant)
> > > > > idea. :-)
> > > >
> > > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
> > > > original thread: are you willing to step up and maintain such a project?
> > >
> > > Yes, I am ("how hard can it be?", he asks himself, knowing all the
> > > while that it's a really bad idea to be asking that question. :-).
> > > But I haven't the faintest idea of how or where to even start, so
> > > pointers would be appreciated.
> >
> > Create/modify a script to automate some kind of download/sync, test and
> > send failure results somewhere. Make it extensible, so that other tests
> > can be easily added -- preferable in a self contained way. It should grow
> > from there.
> >
> > Gavin
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 5: Have you checked our extensive FAQ?
> >
> > http://www.postgresql.org/docs/faqs/FAQ.html
> >
>
>
On Thu, 26 Jun 2003 22:55:45 -0400, Tom Lane <tgl@sss.pgh.pa.us> wrote: >> I've not tried, but if PostgreSQL can do an 'out of tree' compile it >> could make it much easier. > >Yes it can, following the usual procedure for autoconfiscated trees: >just invoke configure from an empty directory, eg > mkdir build > cd build > ../pgsql/configure I do this all the time. Works pretty well. The only minor annoyance is that some files are created in the source tree: ./src/backend/bootstrap/bootparse.c ./src/backend/bootstrap/bootscanner.c ./src/backend/bootstrap/bootstrap_tokens.h ./src/backend/parser/gram.c ./src/backend/parser/parse.h ./src/backend/parser/scan.c ./src/backend/utils/misc/guc-file.c ./src/bin/psql/sql_help.h ./src/interfaces/ecpg/preproc/pgc.c ./src/interfaces/ecpg/preproc/preproc.c ./src/interfaces/ecpg/preproc/preproc.h ./src/pl/plpgsql/src/pl.tab.h ./src/pl/plpgsql/src/pl_gram.c ./src/pl/plpgsql/src/pl_scan.c This didn't itch enough to make me scratch ;-) ServusManfred
Manfred Koizar <mkoi-pg@aon.at> writes:
> I do this all the time. Works pretty well. The only minor annoyance
> is that some files are created in the source tree:
Yeah, that's deliberate: those files are shipped in distribution
tarballs (so that tarball users don't have to have bison/flex/perl
to build from a source tarball). So they need to be made in the
source tree. Since they are platform-independent this isn't any
big problem for the normal uses of out-of-tree building. You could
get rid of them I believe by doing a "make maintainer-clean", but
for testing purposes I suspect that's just a waste of cycles. When
the underlying files haven't changed, there's no reason to regenerate
these files.
regards, tom lane
Tom Lane writes: > This is something that breaks regularly because few of the key > developers use it :-(. If there were automatic tests that used that > build setup, it would be a good thing 'cause it'd keep us honest. This should be included in 'make distcheck'. I'm quite puzzled right now why it doesn't to it already. A DESTDIR installation should also be tested there. Once that is done, ./configure [options] && make distcheck && make maintainer-clean is a quite extensive test. The only drawback is that it is hard to isolate the log of the failing part (config.log, build log, regression test log). For those who don't know, 'make distcheck' builds a distribution, unpacks it, configures, builds, and installs the temporary tree, runs the regression tests, uninstalls, checks whether all files were correctly uninstalled, and builds another distribution out of the temporary tree to check whether the distribution is self-contained. > > That said, poor cleanup would be a result of > > a broken make clean. > > Not to worry, the developers will notice that case fast enough. This is not my experience, but a check for this could be included in make distcheck as well. -- Peter Eisentraut peter_e@gmx.net
Thomas Swan writes: > I just am really concerned about the uninstall/clean up phase and how > that can be done in an orderly fashion. Unless the process can start > from a clean state again, then it won't be valid. The only clean state is if you remove the entire source tree and check it out again. (Of course to save bandwidth, you copy the checked out source tree to a temporary location, do your testing, and then remove that temporary tree.) Relying on make clean or make uninstall is flawed, because those are among the things you want to test. -- Peter Eisentraut peter_e@gmx.net
Manfred Koizar writes: > I do this all the time. Works pretty well. The only minor annoyance > is that some files are created in the source tree: This is intentional, because said files are prebuilt in the distribution, so they always have to be in the source directory. -- Peter Eisentraut peter_e@gmx.net
Bruce Momjian writes: > Amazing you find 688 bytes worth discussing. I know you said "what > happens if everyone adds their scripts", but something that would be a > mess if everyone did it isn't always a proper way to judge if something > is appropriate. I said, if everyone adds their test methodologies. That leads to discrepancies, more of them down the road if one method changes and the other doesn't catch up. For instance, your method just calls pg_ctl, createdb, etc. from the path. If people already have a stable installation of PostgreSQL on their machine, then this will test the wrong installation. So, from now on, if someone submits a test result I have to ask, "which method did you use" -- "don't use that method, because it's wrong". That is one instance, and I'm sure you'll fix it, but there might be more. What I'm saying is, we were in a discussion about improving the testing of PostgreSQL, and this is not a step forward. If we need to improve the testing mechanisms for various purposes -- patch application, automated testing, etc. -- let's look at it and see how we can improve the current infrastructure without inventing a parallel one. At this point, I'm not sure why "make check" doesn't serve you. Perhaps you are not fully aware of what it does (I guess so, from looking at your script). -- Peter Eisentraut peter_e@gmx.net
Peter Eisentraut wrote: >Thomas Swan writes: > > > >>I just am really concerned about the uninstall/clean up phase and how >>that can be done in an orderly fashion. Unless the process can start >>from a clean state again, then it won't be valid. >> >> > >The only clean state is if you remove the entire source tree and check it >out again. (Of course to save bandwidth, you copy the checked out source >tree to a temporary location, do your testing, and then remove that >temporary tree.) Relying on make clean or make uninstall is flawed, >because those are among the things you want to test. > That sounds plausible. Should we let everything stay in the compilers directory. Something like the configure --prefix=$TEST_ROOT and that way we can have the whole thing run as one user in one directory so that system wide impact is minimal. I guess what I'm concerned with is running this on a clean system, and then leaving unknown artifacts behind. Can/does make install output each file it's copying and where to. Capturing that output would make life easier for clean up of things installed outside of the work directory, and provide a more controlled environment.
Wow, I am impressed by 'gmake check'. Who did all that work? It is great. I modified tools/pgtest to use 'gmake check'. Thanks. --------------------------------------------------------------------------- Peter Eisentraut wrote: > Bruce Momjian writes: > > > Amazing you find 688 bytes worth discussing. I know you said "what > > happens if everyone adds their scripts", but something that would be a > > mess if everyone did it isn't always a proper way to judge if something > > is appropriate. > > I said, if everyone adds their test methodologies. That leads to > discrepancies, more of them down the road if one method changes and the > other doesn't catch up. For instance, your method just calls pg_ctl, > createdb, etc. from the path. If people already have a stable > installation of PostgreSQL on their machine, then this will test the wrong > installation. So, from now on, if someone submits a test result I have to > ask, "which method did you use" -- "don't use that method, because it's > wrong". That is one instance, and I'm sure you'll fix it, but there might > be more. What I'm saying is, we were in a discussion about improving the > testing of PostgreSQL, and this is not a step forward. If we need to > improve the testing mechanisms for various purposes -- patch application, > automated testing, etc. -- let's look at it and see how we can improve the > current infrastructure without inventing a parallel one. At this point, > I'm not sure why "make check" doesn't serve you. Perhaps you are not > fully aware of what it does (I guess so, from looking at your script). > > -- > Peter Eisentraut peter_e@gmx.net > > -- Bruce Momjian | http://candle.pha.pa.us pgman@candle.pha.pa.us | (610) 359-1001+ If your life is a hard drive, | 13 Roberts Road + Christ can be your backup. | Newtown Square, Pennsylvania19073
Bruce Momjian wrote:
> See my recent commit of src/tools/pgtest. It might be a good start.
I was wondering if some existing framework, like from the Apache Xalan
package, would be a better point to start from? I hate to say it, Bruce,
but you try to reinvent the wheel by starting with a sled.
Jan
>
> ---------------------------------------------------------------------------
>
> Gavin Sherry wrote:
>> On Thu, 26 Jun 2003, Kevin Brown wrote:
>>
>> > The Hermit Hacker wrote:
>> > > On Wed, 25 Jun 2003, Kevin Brown wrote:
>> > >
>> > > > So...would it make sense to create a gborg project to which people who
>> > > > have written their own test suites can contribute whatever code and data
>> > > > they feel comfortable releasing? As a gborg project, it would be
>> > > > separate from the main PG distribution and would thus have no impact on
>> > > > the build process or anything like that. But at the same time, if there
>> > > > are any ideas on testing that people have had, they could be shared with
>> > > > others through that mechanism.
>> > > >
>> > > > And any tests which prove to be particularly useful could make their way
>> > > > into the PG distribution if people here wish.
>> > > >
>> > > > Of course, like anything else this could be a bad (or perhaps redundant)
>> > > > idea. :-)
>> > >
>> > > It doesn't sound like a bad idea ... but, it pretty much comes down to the
>> > > original thread: are you willing to step up and maintain such a project?
>> >
>> > Yes, I am ("how hard can it be?", he asks himself, knowing all the
>> > while that it's a really bad idea to be asking that question. :-).
>> > But I haven't the faintest idea of how or where to even start, so
>> > pointers would be appreciated.
>>
>> Create/modify a script to automate some kind of download/sync, test and
>> send failure results somewhere. Make it extensible, so that other tests
>> can be easily added -- preferable in a self contained way. It should grow
>> from there.
>>
>> Gavin
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faqs/FAQ.html
>>
>
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #
On Sat, 28 Jun 2003, Jan Wieck wrote: > Bruce Momjian wrote: > > See my recent commit of src/tools/pgtest. It might be a good start. > > I was wondering if some existing framework, like from the Apache Xalan > package, would be a better point to start from? I hate to say it, Bruce, > but you try to reinvent the wheel by starting with a sled. Hey, I take offence at that ... up here in Canada, that sled is faster, dontcha know? especially if we throw those dogs in front of them :)