Re: Page-level version upgrade (was: Block-level CRC checks) - Mailing list pgsql-hackers

Robert Haas wrote:
> > Well, there were quite a number of open issues relating to page
> > conversion:
> >
> > ? ? ? ?o ?Do we write the old version or just convert on read?
> > ? ? ? ?o ?How do we write pages that get larger on conversion to the
> > ? ? ? ? ? new format?
> >
> > As I rember the patch allowed read/wite of old versions, which greatly
> > increased its code impact.
> 
> Oh, for sure there were plenty of issues with the patch, starting with
> the fact that the way it was set up led to unacceptable performance
> and code complexity trade-offs.  Some of my comments from the time:
> 
> http://archives.postgresql.org/pgsql-hackers/2008-11/msg00149.php
> http://archives.postgresql.org/pgsql-hackers/2008-11/msg00152.php
> 
> But the point is that the concept, I think, is basically the right
> one: you have to be able to read and make sense of the contents of old
> page versions.  There is room, at least in my book, for debate about
> which operations we should support on old pages.  Totally read only?
> Set hit bits?  Kill old tuples?  Add new tuples?

I think part of the problem is there was no agreement before the patch
was coded and submitted, and there didn't seem to be much desire from
the patch author to adjust it, nor demand from the community because we
didn't need it yet.

> The key issue, as I think Heikki identified at the time, is to figure
> out how you're eventually going to get rid of the old pages.  He
> proposed running a pre-upgrade utility on each page to reserve the
> right amount of free space.
> 
> http://archives.postgresql.org/pgsql-hackers/2008-11/msg00208.php

Right.  There were two basic approaches to handling a patch that would
expand when upgraded to the new version --- either allow the system to
write the old format, or have a pre-upgrade script that moved tuples so
there was guaranteed enough free space in every page for the new format.
I think we agreed that the later was better than the former, and it was
easy because we don't have any need for that at this time.  Plus the
script would not rewrite every page, just certain pages that required
it.

> I don't like that solution.  If the pre-upgrade utility is something
> that has to be run while the database is off-line, then it defeats the
> point of an in-place upgrade.  If it can be run while the database is
> up, I fear it will need to be deeply integrated into the server.  And
> since we can't know the requirements for how much space to reserve
> (and it needn't be a constant) until we design the new feature, this
> will likely mean backpatching a rather large chunk of complex code,
> which to put it mildly, is not the sort of thing we normally would
> even consider.  I think a better approach is to support reading tuples
> from old pages, but to write all new tuples into new pages.  A
> full-table rewrite (like UPDATE foo SET x = x, CLUSTER, etc.) can be
> used to propel everything to the new version, with the usual tricks
> for people who need to rewrite the table a piece at a time.  But, this
> is not religion for me.  I'm fine with some other design; I just can't
> presently see how to make it work.

Well, perhaps the text I wrote above will clarify that the upgrade
script is only for page expansion --- it is not to rewrite every page
into the new format.

> I think the present discussion of CRC checks is an excellent test-case
> for any and all ideas about how to solve this problem.  If someone can
> get a patch committed than can convert the 8.4 page format to an 8.5
> format with the hint bits shuffled around a (hopefully optional) CRC
> added, I think that'll become the de facto standard for how to handle
> page format upgrades.

Well, yea, the idea would be that the 8.5 server would either convert
the page to the new format on read (assuming there is enough free space,
perhaps requiring a pre-upgrade script), or have the server write the
page in the old 8.4 format and not do CRC checks on the page.  My guess
is the former.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +