Thread: Concerns about this release

I know I have expressed these concerns before but lost the argument, or
at least no one rallied to my position, but I feel I have to mention
these again because they came up during beta.

My first concern is that VACUUM now defaults to the non-locking version.
While I appreciate the new non-locking code, I imagine a release where
non-locking VACUUM will happen automatically, making no need to run
VACUUM.  However, locking VACUUM will still need to be run by
administrators, so we may be left with a VACUUM no one needs to run but
a VACUUM FULL that does need to be run, leaving us with a useless
default for VACUUM without the FULL keyword.  Also, because VACUUM does
not store the freetuple list between postmaster restarts, nor does it
have any way of recording _all_ free tuples, it has to be run with a
different frequency than the old VACUUM that I assume most people ran at
night.  I would have preferred to leave VACUUM as locking VACUUM and
create a new lighter option to the VACUUM command, and if light vacuum
later becomes automatic, the option can just go away.

Second, I am concerned about the removal of oids from system tables.  I
realize this was done to prevent oid usage, particularly by the creation
of temp tables, but such savings only make sense when oids are turned
off in postgresql.conf.  I imagine future releases where we have a
separate oid counter for system/user tables, or 8-bytes oids, in which
case the removal of oids from system tables may be needless.  We have
seen OpenACS is broken now because the new pg_description requires a
separate classoid/objsubid columns to uniquely access tables without
oids, like pg_attribute.  These new columns are used only for non-oid
tables, making access cumbersome, rather than the simpler oid lookup.  I
don't even know if the current setup will allow other tables without
oids to be referenced cleanly.  Object dependency tracking, using
pg_depend, will also require these additional fields to track
dependency, rather than just using the oid, and such access will be more
confusing.

I realize the motivation for these changes were to make PostgreSQL more
enterprise-ready, but I am concerned these changes may need to be
modified in future releases, causing confusion and porting problems for
users.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 

Bruce Momjian wrote:
> 
> I know I have expressed these concerns before but lost the argument, or
> at least no one rallied to my position, but I feel I have to mention
> these again because they came up during beta.
> 
> My first concern is that VACUUM now defaults to the non-locking version.
> While I appreciate the new non-locking code, I imagine a release where
> non-locking VACUUM will happen automatically, making no need to run
> VACUUM.  However, locking VACUUM will still need to be run by
> administrators, so we may be left with a VACUUM no one needs to run but
> a VACUUM FULL that does need to be run, leaving us with a useless
> default for VACUUM without the FULL keyword.  Also, because VACUUM does
> not store the freetuple list between postmaster restarts, nor does it
> have any way of recording _all_ free tuples, it has to be run with a
> different frequency than the old VACUUM that I assume most people ran at
> night.  I would have preferred to leave VACUUM as locking VACUUM and
> create a new lighter option to the VACUUM command, and if light vacuum
> later becomes automatic, the option can just go away.

I kind of second your opinion here. I also have my doubts that the
default is not as well tested as the option. Plus, aren't there some
isses with the non-locking vacuum?

> 
> Second, I am concerned about the removal of oids from system tables.  I
> realize this was done to prevent oid usage, particularly by the creation
> of temp tables, but such savings only make sense when oids are turned
> off in postgresql.conf.  I imagine future releases where we have a
> separate oid counter for system/user tables, or 8-bytes oids, in which
> case the removal of oids from system tables may be needless.  We have
> seen OpenACS is broken now because the new pg_description requires a
> separate classoid/objsubid columns to uniquely access tables without
> oids, like pg_attribute.  These new columns are used only for non-oid
> tables, making access cumbersome, rather than the simpler oid lookup.  I
> don't even know if the current setup will allow other tables without
> oids to be referenced cleanly.  Object dependency tracking, using
> pg_depend, will also require these additional fields to track
> dependency, rather than just using the oid, and such access will be more
> confusing.
> 
> I realize the motivation for these changes were to make PostgreSQL more
> enterprise-ready, but I am concerned these changes may need to be
> modified in future releases, causing confusion and porting problems for
> users.

I don't understand why the oids were taken from the system tables.
Surely there can't be so many that it is a real problem of use. The
dangerous issue is, of course, oid reuse in system tables.

The ability to remove OIDs from user tables is a big bonus. There are so
many times when you just want to store huge amount of lookup data in a
static table, there is no need for the overhead of an OID. This saves
disk space and OID depletion (wrap around).

Are all the PostgreSQL developers really, really, sure that all the new
features in 7.2 are ready for prime time?

Bruce Momjian <pgman@candle.pha.pa.us> writes:
> I know I have expressed these concerns before but lost the argument,

Yes, you did, and it's way too late to bring them up again.
Particularly the OID issue; do you seriously propose an initdb
at this stage to put back OIDs in the system tables?

But for the record:

I think your argument about VACUUM misses the point.  The reason FULL
isn't the default is that we want the default form to be the one people
most want to use.  If lightweight VACUUM starts to be run automatically
in some future release, FULL might at that time become the default.
I don't see anything wrong with changing the default behavior of the
command whenever the system's other behavior changes enough to alter the
"typical" usage of the command.

As for pg_description, the change in primary key is unfortunate but
*necessary*.  I don't foresee us reversing it.  The consensus view as
I recall it was that we wanted to go over to a separate OID generator
per table in some future release, which fits right in with the new
structure of pg_description, but is entirely unworkable with the old.
        regards, tom lane

Bruce Momjian wrote:


> My first concern is that VACUUM now defaults to the non-locking version.
> While I appreciate the new non-locking code, I imagine a release where
> non-locking VACUUM will happen automatically, making no need to run
> VACUUM.  However, locking VACUUM will still need to be run by
> administrators, so we may be left with a VACUUM no one needs to run but
> a VACUUM FULL that does need to be run, leaving us with a useless
> default for VACUUM without the FULL keyword.  Also, because VACUUM does
> not store the freetuple list between postmaster restarts, nor does it
> have any way of recording _all_ free tuples, it has to be run with a
> different frequency than the old VACUUM that I assume most people ran at
> night.  I would have preferred to leave VACUUM as locking VACUUM and
> create a new lighter option to the VACUUM command, and if light vacuum
> later becomes automatic, the option can just go away.


I've kept my mouth shut about this mostly because I've been extremely 
busy and because I've suspected my opinion is a minority one.

In general I think changing the behavior of familiar commands should be 
avoided as much as possible.  VACUUM has worked the same way "since the 
beginning" and this seems like a gratuitous semantic change.

(NOT the new code, but rather the fact that VACUUM now uses the new 
code, and the fact that you need to do a VACUUM FULL to get the old).

It just seems unnecessary.


> Second, I am concerned about the removal of oids from system tables.  I
> realize this was done to prevent oid usage, particularly by the creation
> of temp tables, but such savings only make sense when oids are turned
> off in postgresql.conf.  I imagine future releases where we have a
> separate oid counter for system/user tables, or 8-bytes oids, in which
> case the removal of oids from system tables may be needless.  We have
> seen OpenACS is broken now because the new pg_description requires a
> separate classoid/objsubid columns to uniquely access tables without
> oids, like pg_attribute


This one doesn't bother me particularly.  We were only caught by 
surprise because those of us who follow PG developments haven't been 
paying as close attention as we have in the past.  Mostly because in the 
past we've been desperate for new features and thus working with "the 
next PG beta" version, thus tracking progress and changes in upcoming 
releases has been central to managing our project.  It's a tribute to 
all the hard work you guys have done in the past couple of years that 
we've decided to support PG 7.1 as well as PG 7.2 (for those to chicken 
to change over).

In this case I just wrote a small PL/pgSQL function that queries 
"version()" and generates a view which works properly for the given 
7.1/7.2 version.

I think that's reasonable when you're mucking around system tables.  The 
new function to grab the comment is cleaner, too.

I'm not trying to undercut Bruce's overall argument, just pointing out 
that as the OpenACS 4 project manager I have no strong feelings either 
way.  Bruce, obviously, does and has thought about it a lot more than I 
have.


-- 
Don Baccus
Portland, OR
http://donb.photo.net, http://birdnotes.net, http://openacs.org

mlw <markw@mohawksoft.com> writes:
> I kind of second your opinion here. I also have my doubts that the
> default is not as well tested as the option.

By that logic, we could never make any new releases, or at least never
add any new code.  "New code isn't as well tested as old code" is an
unhelpful observation.

FWIW, I trust lazy VACUUM a lot *more* than I trust the old VACUUM code.
Read the tuple-chain-moving logic in vacuum.c sometime, and then tell me
how confident you feel in it.  (My gut tells me that that logic is
responsible for the recent reports of duplicate tuples in 7.1.*, though
I can't yet back this up with any evidence.)

> Plus, aren't there some isses with the non-locking vacuum?

Such as?

> Are all the PostgreSQL developers really, really, sure that all the new
> features in 7.2 are ready for prime time?

See above.  If you like, we'll push out the release date a few years.
Of course, the code won't get any more ready for prime time just by
sitting on it.

I think that we've very nearly reached the point where further time in
beta phase isn't going to yield any new info.  Only putting it out as
an official release will draw enough new users to expose remaining bugs.
We've been through this same dynamic with every previous release; 7.2
won't be any different.
        regards, tom lane

At 10:58 AM 12/18/01 -0500, mlw wrote:
>
>Are all the PostgreSQL developers really, really, sure that all the new
>features in 7.2 are ready for prime time?

Probably needs more testing - given the recent performance stuff and some
other issues.

For me there's no rush - vacuum isn't as big a pain to me as it is for
other people, and 7.1.3 seems to work well enough for now ;).

7.1 was significantly better overall than 7.0 which was better overall than
6.5.3.

I hope once 7.2 is ready it will be significantly better than 7.1. This
time it might not be better in everything, but hopefully it'll be
convincingly better.

Cheerio,
Link.

Tom Lane wrote:
> 
> mlw <markw@mohawksoft.com> writes:
> > I kind of second your opinion here. I also have my doubts that the
> > default is not as well tested as the option.
> 
> By that logic, we could never make any new releases, or at least never
> add any new code.  "New code isn't as well tested as old code" is an
> unhelpful observation.

Oh come on now. That isn't the point. One could, however, leave the
default "as is" and make the new feature the option for at least one
release cycle. That seem pretty sane without being overly conservative.

> 
> FWIW, I trust lazy VACUUM a lot *more* than I trust the old VACUUM code.
> Read the tuple-chain-moving logic in vacuum.c sometime, and then tell me
> how confident you feel in it.  (My gut tells me that that logic is
> responsible for the recent reports of duplicate tuples in 7.1.*, though
> I can't yet back this up with any evidence.)
> 
> > Plus, aren't there some isses with the non-locking vacuum?
> 
> Such as?

I'm not sure, I have vague recollection of some things (I thought the
duplicate primary keys was on 7.2), but if you think it is good, then
I'll take your word for it.

> 
> > Are all the PostgreSQL developers really, really, sure that all the new
> > features in 7.2 are ready for prime time?
> 
> See above.  If you like, we'll push out the release date a few years.
> Of course, the code won't get any more ready for prime time just by
> sitting on it.


No, that isn't my point. My point is the changes in OIDs and the new
vacuum code seem like a more drastic set of changes than previous
releases. 

Again, there is a time on every project when it is speak now or forever
hold your peace. Bruce spoke, he raised some concerns, I had similar
ones. There can be no harm in doing a little retrospect. 

> 
> I think that we've very nearly reached the point where further time in
> beta phase isn't going to yield any new info.  Only putting it out as
> an official release will draw enough new users to expose remaining bugs.
> We've been through this same dynamic with every previous release; 7.2
> won't be any different.

I agree.

Tom Lane wrote:

> mlw <markw@mohawksoft.com> writes:
> 
>>I kind of second your opinion here. I also have my doubts that the
>>default is not as well tested as the option.
>>
> 
> By that logic, we could never make any new releases, or at least never
> add any new code.  "New code isn't as well tested as old code" is an
> unhelpful observation.


I'd switch production sites I control more quickly if I didn't have to 
run around and change scripts galore to say "VACUUM FULL" rather than 
"VACUUM".  I personally will let the new VACUUM code run on non-critical 
sites for a few months before using it on production sites.

The issue isn't new code, the issue is changing semantics for an old 
command when there is no need to do so.

That is a very different thing.

One such change is no big deal, but it's a bad product design philosphy 
in general.

-- 
Don Baccus
Portland, OR
http://donb.photo.net, http://birdnotes.net, http://openacs.org

mlw <markw@mohawksoft.com> writes:

> Tom Lane wrote:
> > 
> > mlw <markw@mohawksoft.com> writes:
> > > I kind of second your opinion here. I also have my doubts that the
> > > default is not as well tested as the option.
> > 
> > By that logic, we could never make any new releases, or at least never
> > add any new code.  "New code isn't as well tested as old code" is an
> > unhelpful observation.
> 
> Oh come on now. That isn't the point. One could, however, leave the
> default "as is" and make the new feature the option for at least one
> release cycle. That seem pretty sane without being overly
> conservative.

Actually I think that it makes a lot of sense to change the semantics
of VACUUM.  None of us here like the way that vacuum currently works.
The PostgreSQL hackers could have given the new VACUUM a different
name, but a million email messages in the mailing list archives tell
newbies to VACUUM.  That's why changing VACUUM so that it "does what
you mean" makes perfect sense.  I am looking forward to being able to
vacuum my database while the plant is still running.

For most people (especially the new PostgreSQL users) the new VACUUM
is precisely what they want.  They don't want to lock their tables
while vacuuming.  The fact that Tom trusts the new vacuum over the old
one should clinch the matter.

Those of you that have 7.1 databases in production that don't mind the
current VACUUM implementation can simply wait until the new VACUUM
gets put out in production and see how it goes.  If you are happy with
what you have now their is little pressure to upgrade.  However,
changing the default makes it easier for new PostgreSQL users.  It
also means that the PostgreSQL hackers can say that their database is
ready for 24x7 operation in "the default" mode.

Advertising is important too.

> > FWIW, I trust lazy VACUUM a lot *more* than I trust the old VACUUM
> > code.  Read the tuple-chain-moving logic in vacuum.c sometime, and
> > then tell me how confident you feel in it.  (My gut tells me that
> > that logic is responsible for the recent reports of duplicate
> > tuples in 7.1.*, though I can't yet back this up with any
> > evidence.)
> > 
> > > Plus, aren't there some isses with the non-locking vacuum?
> > 
> > Such as?
> 
> I'm not sure, I have vague recollection of some things (I thought the
> duplicate primary keys was on 7.2), but if you think it is good, then
> I'll take your word for it.
> 
> > 
> > > Are all the PostgreSQL developers really, really, sure that all the new
> > > features in 7.2 are ready for prime time?
> > 
> > See above.  If you like, we'll push out the release date a few years.
> > Of course, the code won't get any more ready for prime time just by
> > sitting on it.
> 
> 
> No, that isn't my point. My point is the changes in OIDs and the new
> vacuum code seem like a more drastic set of changes than previous
> releases.

I think that you are forgetting some of PostgreSQL's past changes.
Take a look at the changelog and you will realize that removing OIDs
and changine the semantics of VACUUM are peanuts compared to 7.1's
transaction log, TOAST, Outer Joins, the new function manager, etc.

The developers have been warning against using OIDs forever, and you
can still leave them on if you need them.  And the new vacuum is a
vast improvement for those of us that use our databases 24x7.  I see
this as a bug fix of the old procedure with the option of using the
old implementation if you really need it.

> Again, there is a time on every project when it is speak now or
> forever hold your peace. Bruce spoke, he raised some concerns, I had
> similar ones. There can be no harm in doing a little retrospect.

That's certainly true.  I suppose the primary difference between my
attitude and the yours and Bruce's is that I can't hardly wait for the
new features in 7.2 :).  Likewise you don't remember the massive
changes in 7.1 because you *needed* them.  Now you are happy with 7.1
and don't want to make radical changes :).

> > I think that we've very nearly reached the point where further time in
> > beta phase isn't going to yield any new info.  Only putting it out as
> > an official release will draw enough new users to expose remaining bugs.
> > We've been through this same dynamic with every previous release; 7.2
> > won't be any different.
> 
> I agree.

Bring it on!  I have good backups :).

Jason

Tom Lane wrote:
> 
> mlw <markw@mohawksoft.com> writes:
> > I kind of second your opinion here. I also have my doubts that the
> > default is not as well tested as the option.
> 
> By that logic, we could never make any new releases, or at least never
> add any new code.  "New code isn't as well tested as old code" is an
> unhelpful observation.
> 
> FWIW, I trust lazy VACUUM a lot *more* than I trust the old VACUUM code.
> Read the tuple-chain-moving logic in vacuum.c sometime, and then tell me
> how confident you feel in it.  (My gut tells me that that logic is
> responsible for the recent reports of duplicate tuples in 7.1.*, though
> I can't yet back this up with any evidence.)

For all the various bugs which have been in PG over the years, the
recent crop of "duplicate tuples" is the absolute scariest. Can a
release really be made without knowing precisely the cause of those
corruptions? The various theories offered by the posters (SMP
machine, CREATE INDEX in pl/pgsql functions, etc.) aren't comforting
either. To me, all other feature enhancements pale in comparison to
pinning down this bug.

Just my opinion, 

Mike Mascari
mascarm@mascari.com

Mike Mascari <mascarm@mascari.com> writes:

> Tom Lane wrote:
> > 
> > FWIW, I trust lazy VACUUM a lot *more* than I trust the old VACUUM code.
> > Read the tuple-chain-moving logic in vacuum.c sometime, and then tell me
> > how confident you feel in it.  (My gut tells me that that logic is
> > responsible for the recent reports of duplicate tuples in 7.1.*, though
> > I can't yet back this up with any evidence.)
> 
> For all the various bugs which have been in PG over the years, the
> recent crop of "duplicate tuples" is the absolute scariest. Can a
> release really be made without knowing precisely the cause of those
> corruptions? The various theories offered by the posters (SMP
> machine, CREATE INDEX in pl/pgsql functions, etc.) aren't comforting
> either. To me, all other feature enhancements pale in comparison to
> pinning down this bug.

The one instance of the bug that has been definitely pinned down
involved the old VACUUM code in 7.1.3 (plus a don't-do-that-then index
function).

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.  --T. J. Jackson, 1863

Mike Mascari <mascarm@mascari.com> writes:
> For all the various bugs which have been in PG over the years, the
> recent crop of "duplicate tuples" is the absolute scariest. Can a
> release really be made without knowing precisely the cause of those
> corruptions?

I'm not happy about it either, but the existence of an unrepaired bug
in 7.1 is hardly grounds for not releasing 7.2.  Finding a bug in 7.2
that doesn't exist in 7.1 would be such grounds, sure.  But we put out
releases when we think they are better than what's out there; holding
them in a (vain) quest for perfection isn't the way to make progress.

FWIW, we seem to understand the mechanism behind Brian Hirt's case,
at least.  Need to poll the other reporters and see if the explanation
fits their cases.
        regards, tom lane

Don Baccus <dhogaza@pacifier.com> writes:
> 5. The semantics of VACUUM have changed.  Silently (in the sense that
>     there's no notification or warning spewed out).

???  VACUUM has no semantics: it does not alter your data (or at least
it's not supposed to ;-)).  This change is transparent in the same
way that the WAL and function manager changes were.  If there is any
lack of transparency, it would show up as greater disk space usage
than you might expect --- which seems *exactly* parallel to WAL.
And you don't have the option to turn WAL off.  I don't think you
can consistently maintain that adding WAL is good and changing VACUUM
is bad.

> Or if the new one hoses your tables.   Tom's bright, but he's not 
> certified bug-free.

Certainly, but you are assuming that the old VACUUM is bug-free,
which is, um, overly optimistic.  The new VACUUM code is (mostly) a
subset of the old, and has removed all the most ticklish bits of the old
code.  So if you are looking for the fewest bugs you should prefer the
new to the old anyway.  Case in point: Brian Hirt's bug does not arise
under new-style VACUUM.  I had to say VACUUM FULL to make it happen
in current sources.
        regards, tom lane

mlw <markw@mohawksoft.com> writes:
> Plus, aren't there some isses with the non-locking vacuum?
>> 
>> Such as?

> I'm not sure, I have vague recollection of some things (I thought the
> duplicate primary keys was on 7.2), but if you think it is good, then
> I'll take your word for it.

All the reports of duplicate rows are from people running 7.1.*.
The bug (or bugs) probably do exist in current sources as well,
but you cannot blame non-locking vacuum for it.
        regards, tom lane

Jason Earl wrote:


> 
> Actually I think that it makes a lot of sense to change the semantics
> of VACUUM.  None of us here like the way that vacuum currently works.


None of us?  You forgot to ask me when you polled all of us ...

I think the new VACUUM strategy has great promise, but until I see it 
running in production environments for a while that's as far as I'm 
willing to go.  I've lived with the old VACUUM for the two or so years 
I've been running PG-based websites, I can live with it a few more 
months until you run into all the table-corrupting bugs for me.  Or have 
such a successful experience that I become convinced there aren't any 
(my hope, of course!)

> It also means that the PostgreSQL hackers can say that their database is
> ready for 24x7 operation in "the default" mode.


Plenty of people already run it 24x7.


> I think that you are forgetting some of PostgreSQL's past changes.
> Take a look at the changelog and you will realize that removing OIDs
> and changine the semantics of VACUUM are peanuts compared to 7.1's
> transaction log, TOAST, Outer Joins, the new function manager, etc.


While true, it is also true that:

1. TOAST didn't change the semantics of queries on existing tables.  It   just allows you to define tables that weren't
definablebefore.
 

2. outer joins didn't change the semantics of existing queries.  You can   write queries you couldn't write before, but
yourold queries work   without change.
 

3. The new function manager is transparent to all but those who write C   functions, other than fixing the NULL
parameterbug.
 

4. WAL was a scary change due to the scope, but it didn't change the   semantics of transactions.

5. The semantics of VACUUM have changed.  Silently (in the sense that   there's no notification or warning spewed
out).

> And the new vacuum is a
> vast improvement for those of us that use our databases 24x7.  I see
> this as a bug fix of the old procedure with the option of using the
> old implementation if you really need it.


Or if the new one hoses your tables.   Tom's bright, but he's not 
certified bug-free.

>>Again, there is a time on every project when it is speak now or
>>forever hold your peace. Bruce spoke, he raised some concerns, I had
>>similar ones. There can be no harm in doing a little retrospect.
>>
> 
> That's certainly true.  I suppose the primary difference between my
> attitude and the yours and Bruce's is that I can't hardly wait for the
> new features in 7.2 :).  Likewise you don't remember the massive
> changes in 7.1 because you *needed* them.  Now you are happy with 7.1
> and don't want to make radical changes :).


I think the new VACUUM is a great thing, and frankly I don't care all 
that much about the VACUUM FULL command issue.  I tend to take the view 
that command semantics shouldn't be changed if one can avoid it.  It's a 
view built on the fact that about twenty years of my past life were 
spent as a language implementor and dealing with customer feedback when 
I've been foolish enough to make semantics changes of this sort.

I certainly wouldn't've raised this issue, at this late date, on my own.

But since Bruce did and since I'm temporarily subscribed to the list I 
feel justified in piling on :)


> Bring it on!  I have good backups :).


Uh ... if you're truly 24x7 critical then rolling back to your last 
backup wouldn't cut it, I'd think  ...



-- 
Don Baccus
Portland, OR
http://donb.photo.net, http://birdnotes.net, http://openacs.org

> Bruce Momjian <pgman@candle.pha.pa.us> writes:
> > I know I have expressed these concerns before but lost the argument,
> 
> Yes, you did, and it's way too late to bring them up again.
> Particularly the OID issue; do you seriously propose an initdb
> at this stage to put back OIDs in the system tables?

No, I don't expect any changes.  I just felt I needed to clearly state
my opinion on this so people know where we are headed.

> But for the record:
> 
> I think your argument about VACUUM misses the point.  The reason FULL
> isn't the default is that we want the default form to be the one people
> most want to use.  If lightweight VACUUM starts to be run automatically
> in some future release, FULL might at that time become the default.
> I don't see anything wrong with changing the default behavior of the
> command whenever the system's other behavior changes enough to alter the
> "typical" usage of the command.

Agreed.  VACUUM nolock will be the most used method of vacuum for 7.2.

My concern was that FULL is still needed sometimes _and_ may become the
more popular vacuum method in later releases as vacuum nolock becomes
automatic.  

Vacuum may go from locking (7.1), to non-locking (7.2), to locking (7.3)
in the span of three releases.  My point was that keeping vacuum as
locking in all releases and adding a non-locking version only for 7.2
seemed cleaner.

Now, if you think we will continue needing a non-locking vacuum in all
future releases then we are doing the right thing by making non-locking
the default.  Is that true?

> As for pg_description, the change in primary key is unfortunate but
> *necessary*.  I don't foresee us reversing it.  The consensus view as
> I recall it was that we wanted to go over to a separate OID generator
> per table in some future release, which fits right in with the new
> structure of pg_description, but is entirely unworkable with the old.

In other words, a separate sequence for each system table, right?  Is
that where we are headed?  We could still call the column oid and
queries would continue to work.  I don't think there are many
cases where the oid is used to find a particular table, except my
/contrib/findoidjoins, which can be removed.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 

Bruce Momjian wrote:

>>As for pg_description, the change in primary key is unfortunate but
>>*necessary*.  I don't foresee us reversing it.  The consensus view as
>>I recall it was that we wanted to go over to a separate OID generator
>>per table in some future release, which fits right in with the new
>>structure of pg_description, but is entirely unworkable with the old.
>>
This is the clash of views between OO and R parts of ORDB - tho OO part
_needs_ oid and a better support structure for OIDs, while the classical 
RDB
(aka. bean-counting ;) part has not need for them..

It's not the right time to discuss it during beta time, but the pendulum 
has been
swinging heavily to the "classic RDB" side of thing in recent years for 
PostgreSQL,
while it has been already going the other way for at least Oracle and 
Informix(Illustra)
at least.

If we want to keep up the general copycat image of free software we of 
course can,
but I would much more like us to "return to the roots" of postgres and 
be in/near the
forefront for a change ;) . That would mean at least stronger support 
for OO, and
perhaps also restoring support for (per table/optional) time travel.

There were possibly more nice features that could be restored...

>
>In other words, a separate sequence for each system table, right?  Is
>that where we are headed?  We could still call the column oid and
>queries would continue to work.  I don't think there are many
>cases where the oid is used to find a particular table, except my
>/contrib/findoidjoins, which can be removed.
>
In the 7.x.x series even a system column tableoid was added that can be 
used to
determine the table the tuple came form. I guess it was in preparation 
for implementing
SQL99's CREATE TABLE x UNDER y construct in one table, perhaps in a way that
would allow fast DROP COLUMN's (i.e separation of physical/logical 
column order)

----------------
Hannu

Hannu Krosing <hannu@tm.ee> writes:
> This is the clash of views between OO and R parts of ORDB - tho OO part
> _needs_ oid and a better support structure for OIDs, while the classical 
> RDB (aka. bean-counting ;) part has not need for them..

What's that have to do with it?  The direction we are moving in is that
the globally unique identifier of an object is tableoid+rowoid, not just
oid; but I fail to see why that's less support than before.  If
anything, I think it's better support.  The tableoid tells you which
table the object is in, and thus its type, whereas a single global OID
sequence gives you no information at all about what the object
represented by an OID is or where to look for it.
        regards, tom lane

> Hannu Krosing <hannu@tm.ee> writes:
> > This is the clash of views between OO and R parts of ORDB - tho OO part
> > _needs_ oid and a better support structure for OIDs, while the classical 
> > RDB (aka. bean-counting ;) part has not need for them..
> 
> What's that have to do with it?  The direction we are moving in is that
> the globally unique identifier of an object is tableoid+rowoid, not just
> oid; but I fail to see why that's less support than before.  If
> anything, I think it's better support.  The tableoid tells you which
> table the object is in, and thus its type, whereas a single global OID
> sequence gives you no information at all about what the object
> represented by an OID is or where to look for it.

I like that idea a lot.  I had not see that proposed before, to use a
combination table oid/sequence as the globally unique oid.  Nice.

--  Bruce Momjian                        |  http://candle.pha.pa.us pgman@candle.pha.pa.us               |  (610)
853-3000+  If your life is a hard drive,     |  830 Blythe Avenue +  Christ can be your backup.        |  Drexel Hill,
Pennsylvania19026
 

> > Are all the PostgreSQL developers really, really, sure that all the new
> > features in 7.2 are ready for prime time?
>
> See above.  If you like, we'll push out the release date a few years.
> Of course, the code won't get any more ready for prime time just by
> sitting on it.
>
> I think that we've very nearly reached the point where further time in
> beta phase isn't going to yield any new info.  Only putting it out as
> an official release will draw enough new users to expose remaining bugs.
> We've been through this same dynamic with every previous release; 7.2
> won't be any different.

Just for the record, I *never* use a x.x version of anything, especially
Postgres.  I'll wait until at least 7.2.1 before running it on a production
server...

Not saying that there's anything wrong with 7.2 - just that there's _always_
interesting problems (look at the changelogs for 7.1.1 - 7.1.3)

Chris

> I think the new VACUUM is a great thing, and frankly I don't care all
> that much about the VACUUM FULL command issue.  I tend to take the view
> that command semantics shouldn't be changed if one can avoid it.  It's a
> view built on the fact that about twenty years of my past life were
> spent as a language implementor and dealing with customer feedback when
> I've been foolish enough to make semantics changes of this sort.

I actually just add 'vacuumdb -a -z -q' sort of thing to my crontab.  Which
version of VACCUM will vacuumdb be using in 7.2?  Will there be an option to
vacuumdb to run VACUUM FULL if it defaults to just VACUUM?

Chris

"Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:

> Just for the record, I *never* use a x.x version of anything, especially
> Postgres.  I'll wait until at least 7.2.1 before running it on a production
> server...
> 
> Not saying that there's anything wrong with 7.2 - just that there's _always_
> interesting problems (look at the changelogs for 7.1.1 - 7.1.3)

On the other hand, *somebody's* got to use x.x, otherwise the
"interesting" problems would never get found.  ;)

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.  --T. J. Jackson, 1863

>>>Don Baccus said:> > It also means that the PostgreSQL hackers can say that their database is> > ready for 24x7
operationin "the default" mode.> > > Plenty of people already run it 24x7.
 

I guess many do, actually. With the old (existing) code. I used to run most of 
my production systems with beta code years ago - as at that time PostgreSQL 
was beta quality anyway :) - but I currently run 7.0 on such systems, planning 
an upgarde to 7.1 anytime now - actually, the delay is because I need to shut 
down that large 24x7 setup!
> 2. outer joins didn't change the semantics of existing queries.  You can>     write queries you couldn't write
before,but your old queries work>     without change.
 

Not true!

SQL that worked in 7.0 does not work always in 7.1. Arguably, that can be 
explained by 'bad SQL', but does not change the fact.

The default SQL inheritance also makes things much, much different. Let me 
explain with this 'intuitive' schema:

CREATE TABLE maillog (from    text,to    text,msgdate    datetime,msgid    text,bytes    int
);

Now this is the 'current' log data, and we need to archive it periodically, to 
keep the old data and at the same time avoid unnecessary performance decrease. 
So we create archive tables like this

CREATE TABLE maillog200101 () INHERITS (maillog);
CREATE TABLE maillog200102 () INHERITS (maillog);

Data gets there by

INSERT INTO maillog200101 SELECT * FROM maillog WHERE msgdate >= '2001-01-01' 
and msgdate < '2001-02-01';
DELETE FROM maillog WHERE msgdate >= '2001-01-01' and msgdate < '2001-02-01';

etc.

One of the reasons in doing this is that the archive creation software needs 
not know the exact structure of the table it archives...

Under 7.0 everything is as expected. Under 7.1 when you SELECT from maillog, 
you also select from all inherited tables, which is not expected. One can of 
course rewrite all the SELECTs :-)

Sorry for the off-topic, but this hit me badly :)
> 4. WAL was a scary change due to the scope, but it didn't change the>     semantics of transactions.

But it was discussed many, many times before being implemented. 
> 5. The semantics of VACUUM have changed.  Silently (in the sense that>     there's no notification or warning spewed
out).

You argue with Tom here, but both of you go call the same beast differently.
Sure, semantics of the 'VACUUM' command has changed. The old semantics is now 
available as 'VACUUM FULL' - as it was already commented, this is no worry for 
new users. The new semantics of VACUUM may actually be an improvement, because 
many people need to run VACUUM in order to update statistics. The only benefit 
I see from VACUUM FULL is disk space usage reduction - which most 
administrators do not do as default action (that is, rarely in scripts).

If free heap is reused in both tables and indexes, I do not see much trouble 
in the new semantics, especially for 24x7 setups! Tables, that are used for 
'logs' always grow - if you take my example above, given you have ca 1 million 
records a month, after the first month the maillog table is supposed to 
contain 1 million records, then those get deleted and the new tuples will 
occupy another 1 million records disk space... now given that we archive the 
old data and not just throw it away, if free space reuse is available, the 
maillog table will never grow more than 2 million records in size, which is 
reasonable. With the old vacuum semantics, the vacuuming of such tables is 
expensive (as they usually have indexes) and the saved disk space is not 
significant.
> I think the new VACUUM is a great thing, and frankly I don't care all > that much about the VACUUM FULL command
issue. I tend to take the view > that command semantics shouldn't be changed if one can avoid it.  It's a > view built
onthe fact that about twenty years of my past life were > spent as a language implementor and dealing with customer
feedbackwhen > I've been foolish enough to make semantics changes of this sort.
 

I fully agree with this.

However, what I think Tom had not said in his defense is that the new VACUUM 
semantics is expected to greatly improve the experience with PostgreSQL. As 
such, NEW users are more likely to appreciate it. More users = more found bugs 
= better PostgreSQL (perhaps other benefits :) It may also be an improvement 
for true 24x7 setups - users of my PostgreSQL powered company management 
database start to complain when i do VACUUM during the day, as table locking 
is 'freezing' for example cash register operation, and 'people in the queue 
are waiting' etc ;)
> > Bring it on!  I have good backups :).> > > Uh ... if you're truly 24x7 critical then rolling back to your last >
backupwouldn't cut it, I'd think  ...
 

Ah... backups.. and they also load back so slow :)


Daniel

Doug McNaught <doug@wireboard.com> writes:

> "Christopher Kings-Lynne" <chriskl@familyhealth.com.au> writes:
> 
> > Just for the record, I *never* use a x.x version of anything, especially
> > Postgres.  I'll wait until at least 7.2.1 before running it on a production
> > server...
> > 
> > Not saying that there's anything wrong with 7.2 - just that there's _always_
> > interesting problems (look at the changelogs for 7.1.1 - 7.1.3)
> 
> On the other hand, *somebody's* got to use x.x, otherwise the
> "interesting" problems would never get found.  ;)

It's simple.  The people who use the x.x releases are those that
either 1) need the added functionality or 2) haven't read the HACKERS
mailing list :).

Personally the new non-locking VACUUM is enough of a win for me that I
will be giving 7.2 a whirl sooner rather than later.  In fact, I have
a planned downtime over the Christmas holiday, and I have been toying
with the idea of rolling out whatever is available (even if it isn't
the official release).  In fact, if there was a Debian package of
7.2b4 I would probably be running it right now (hint hint).

Jason

Jason Earl wrote:

> In fact, if there was a Debian package of
> 7.2b4 I would probably be running it right now (hint hint).


I know there will be folks putting up demo OpenACS 4 sites on PG 7.2 
something, as soon as we can resolve the problems that prevent it from 
working.   Just because I personally feel cautious about using the new 
VACUUM on production sites doesn't mean I won't be using it on 
non-critical sites ASAP.



-- 
Don Baccus
Portland, OR
http://donb.photo.net, http://birdnotes.net, http://openacs.org

> I know there will be folks putting up demo OpenACS 4 sites on PG 7.2
> something, as soon as we can resolve the problems that prevent it from
> working.   Just because I personally feel cautious about using the new
> VACUUM on production sites doesn't mean I won't be using it on
> non-critical sites ASAP.

I've been trying to follow the VACUUM concern thread but have missed a few
so I thought I might ask someone to summarize the concern and the real risk
of table corruption and any other problems that have been reported... I was
planning on upgrading several production databases to 7.2 the day it was
released (I've been developing with the betas with little problem) but won't
do so if there are some real concerns about corruption and such...

So if anyone doesn't mind to take a minute, could I get opinions? Is it too
paranoid to not use the 7.2 release in production?

Thanks!

-Mitch

"Mitch Vincent" <mitch@doot.org> writes:
> So if anyone doesn't mind to take a minute, could I get opinions? Is it too
> paranoid to not use the 7.2 release in production?

Don is working from the "don't be a pioneer" theory, which is hard to
dispute in the abstract.  In the concrete, though, I see little reason
to think that 7.2 will be less reliable than 7.1.*, even before we fix
the inevitable early-return bugs and issue a 7.2.1.  We have not made
any huge changes like WAL in this cycle.

As an idle exercise, I just went through the CVS log entries since
7.2beta2 (Nov 6, about six weeks ago).  I counted 68 log entries that
I could classify as bug fixes; of these, 47 were for bugs that exist in
7.1, the other 21 for new bugs introduced in 7.2 code.  I'd call about
4 of the old bugs and 6 of the new ones significant issues (eg, a core
dump is significant, fixing to_char's handling of roman numeral dates
is not).  4 out of the 6 significant new-bug fixes were in the first two
weeks of the six-week period.

You can read those numbers however you want, but to me they look like
7.2.0 will be better than 7.1.anything.
        regards, tom lane

"Mitch Vincent" <mitch@doot.org> writes:

> I've been trying to follow the VACUUM concern thread but have missed a few
> so I thought I might ask someone to summarize the concern and the real risk
> of table corruption and any other problems that have been reported... I was
> planning on upgrading several production databases to 7.2 the day it was
> released (I've been developing with the betas with little problem) but won't
> do so if there are some real concerns about corruption and such...
> 
> So if anyone doesn't mind to take a minute, could I get opinions? Is it too
> paranoid to not use the 7.2 release in production?

Interestingly enough, the one definite table-corrupting bug that has
been found was actually in old code (what is now VACUUM FULL).  It was
hard to trigger which explains why it didn't turn up earlier, but it
was hiding in 7.1.x the whole time. 

I will probably put 7.2rc1 on my dev server when it comes out.
I am running a mix of 7.1.x versions in production so I will probably 
migrate them all to 7.1.3 first (because pg_dump in earlier 7.1.x
makes unrestorable dumps in some situations) then look to go to 7.2
after it's been out a bit.

Your call, but I'd certainly run 7.2 on a test box for a little while
before rolling it out anywhere important.

Just my take...

-Doug
-- 
Let us cross over the river, and rest under the shade of the trees.  --T. J. Jackson, 1863

Tom Lane wrote:

> "Mitch Vincent" <mitch@doot.org> writes:
> 
>>So if anyone doesn't mind to take a minute, could I get opinions? Is it too
>>paranoid to not use the 7.2 release in production?
>>
> 
> Don is working from the "don't be a pioneer" theory, which is hard to
> dispute in the abstract.  In the concrete, though, I see little reason
> to think that 7.2 will be less reliable than 7.1.*, even before we fix
> the inevitable early-return bugs and issue a 7.2.1.  We have not made
> any huge changes like WAL in this cycle.


Only on production boxes, though.  In practice, Tom's probably right 
about the relative reliability but, on the other hand ...

This is really the first release cycle where those of us working on the 
OpenACS project feel we have had the luxury of not being early adaptors.  It's a nice feeling for a change and a
reflectionon PG's maturity.
 

Our first version of OpenACS was released to run on PG 6.5 beta.  6.4, 
which sync'd the log file after every select statement (remember those 
days?), just didn't cut it.  We also were crashing the backend in 6.4 
right-and-left and 6.5 fixed a ton of those.  PG 7.0 fixed a bunch of 
bugs that were hindering us and we quickly told everyone using our 
toolkit to upgrade ASAP.  When we inherited the aD code base for what's 
now OpenACS 4 and decided to support both Oracle and PG with common 
source code, we quickly decided we'd only support PG 7.1, with its outer 
joins, even though it was still in a pre-release state.

So ... now I can afford to be conservative and hey, I'm enjoying it!


> You can read those numbers however you want, but to me they look like
> 7.2.0 will be better than 7.1.anything.


None of my sites seem to be affected by any PG 7.1 bug, and as they run 
the same stereotyped queries over and over again, I'm not terribly 
concerned about suddenly hitting a roadbump.

So while PG 7.2 may have fewer bugs than PG 7.1, it may still introduce 
new bugs that will affect me in a negative way.  Just because I appear 
to be dodging 7.1's bullets doesn't mean I'll dodge new rounds fired by 7.2.

Also fixing bugs may expose accidental dependencies on such bugs.  When 
we first started moving OpenACS 3.x from 6.5 to 7.0 we found a few 
illegal queries that had slipped through 6.5, appearing to work, which 
triggered explicit errors in PG 7.0.

So testing first on a development box isn't just a protection against 
any new bugs you introduce.  New fixes can lead to work, too, and it's 
really nicer to find this out on test, not production, boxes.

-- 
Don Baccus
Portland, OR
http://donb.photo.net, http://birdnotes.net, http://openacs.org