Thread: pg_upgrade project status

pg_upgrade project status

From
Zdenek Kotala
Date:
In the last week community made decision about pg_upgrade project and its 
implementation. I would like to try summarize this conclusion and I add other 
topic which should be finished for 8.4.

Convert on read has been selected as a good way, because it is not invasive and 
does not limit fresh database. But, this way needs core modification which 
allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 
will be implemented. Sorry about that, but we need move forward and there is not 
easy way without core modification to do it.

As I mentioned manytimes before there are two major issues with convert on read 
and one small issue.

1) Data does not fit on the new page. It will be solve by pre-upgrade check 
which reserve space on each page, before upgrade.

2) No information about TOASTed data in TOAST table. It is problem for example 
when varlena encoding is changed, or heaptupleheader or array data stucture. 
Idea is to add column position into each chunk and also replace chunk ID by data 
offset.

3) Access methods stores "shadow" data into pages - for example metapages or 
hash index has bitmap pages, but there is no information on the page about this 
and page conversion could lost the data. We need mark this pages. Metapage is 
usually first page and it could be hardwired, but hash bitmap can be everywhere.

I suggest to add page flag HAS_EXTRA_CONTENT (give me a better name).

IIRC, somebody suggest to replace HASH bitmap pages with FSM? Any idea?


I hope that I mention all. Please feel free to add more information or correct 
me if I wrong.
    Thanks Zdenek



Re: pg_upgrade project status

From
Bruce Momjian
Date:
Zdenek Kotala wrote:
> In the last week community made decision about pg_upgrade project and its 
> implementation. I would like to try summarize this conclusion and I add other 
> topic which should be finished for 8.4.
> 
> Convert on read has been selected as a good way, because it is not invasive and 
> does not limit fresh database. But, this way needs core modification which 
> allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 
> will be implemented. Sorry about that, but we need move forward and there is not 
> easy way without core modification to do it.
> 
> As I mentioned manytimes before there are two major issues with convert on read 
> and one small issue.
> 
> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
> which reserve space on each page, before upgrade.

Rather than specifying free space as an amount, I was thinking of having
a boolean like 'ready_for_upgrade' and the system internally would know
how much free space for each page and tuple.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +


Re: pg_upgrade project status

From
Gregory Stark
Date:
Bruce Momjian <bruce@momjian.us> writes:

>> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
>> which reserve space on each page, before upgrade.
>
> Rather than specifying free space as an amount, I was thinking of having
> a boolean like 'ready_for_upgrade' and the system internally would know
> how much free space for each page and tuple.

I think the original idea was that 8.5 would come with a tool which connected
to an 8.4 database and makes sure things are ok. To do so it has to do two
things: 1) set some server state so that the server doesn't create any new
"bad" pages and 2) check that all the existing pages and fix any "bad" pages.

The question here is what state does the server need to have to ensure it
doesn't create any new "bad" pages. The simplest option would be a
"minimum_free_space_per_page" guc variable. That would be sufficient if we're
worried about expanding the page header. The user wouldn't have to know about
these, the tool would set it for him.

If we're worried about expanding tuple header overhead then we would need a
separate option. If we grow any data type representations then we could still
have a problem.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!


Re: pg_upgrade project status

From
Bruce Momjian
Date:
Gregory Stark wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> 
> >> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
> >> which reserve space on each page, before upgrade.
> >
> > Rather than specifying free space as an amount, I was thinking of having
> > a boolean like 'ready_for_upgrade' and the system internally would know
> > how much free space for each page and tuple.
> 
> I think the original idea was that 8.5 would come with a tool which connected
> to an 8.4 database and makes sure things are ok. To do so it has to do two
> things: 1) set some server state so that the server doesn't create any new
> "bad" pages and 2) check that all the existing pages and fix any "bad" pages.
> 
> The question here is what state does the server need to have to ensure it
> doesn't create any new "bad" pages. The simplest option would be a
> "minimum_free_space_per_page" guc variable. That would be sufficient if we're
> worried about expanding the page header. The user wouldn't have to know about
> these, the tool would set it for him.
> 
> If we're worried about expanding tuple header overhead then we would need a
> separate option. If we grow any data type representations then we could still
> have a problem.

Yes, this is why I was thinking it should just be a boolean and the old
server will know the requirements, but it does require us to force a
minor upgrade on the old server so it has the proper information
embedded in the binary.

We could force each page to have the required amount of free space but
the binary is going to need to not invalidate that as database
processing continues before the upgrade-in-place.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +


Re: pg_upgrade project status

From
Zdenek Kotala
Date:
Bruce Momjian napsal(a):
> Zdenek Kotala wrote:
>> In the last week community made decision about pg_upgrade project and its 
>> implementation. I would like to try summarize this conclusion and I add other 
>> topic which should be finished for 8.4.
>>
>> Convert on read has been selected as a good way, because it is not invasive and 
>> does not limit fresh database. But, this way needs core modification which 
>> allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 
>> will be implemented. Sorry about that, but we need move forward and there is not 
>> easy way without core modification to do it.
>>
>> As I mentioned manytimes before there are two major issues with convert on read 
>> and one small issue.
>>
>> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
>> which reserve space on each page, before upgrade.
> 
> Rather than specifying free space as an amount, I was thinking of having
> a boolean like 'ready_for_upgrade' and the system internally would know
> how much free space for each page and tuple.

You need booth, flag which shows you that the relation/database is ready for 
upgrade and free space reservation configuration for each column. System cannot 
know it because PostgreSQL is not oracle :-). It does not know what will happend 
during next version development :-). It have to be setup by pre-upgrade script.

    Zdenek


Re: pg_upgrade project status

From
Bruce Momjian
Date:
Zdenek Kotala wrote:
> Bruce Momjian napsal(a):
> > Zdenek Kotala wrote:
> >> In the last week community made decision about pg_upgrade project and its 
> >> implementation. I would like to try summarize this conclusion and I add other 
> >> topic which should be finished for 8.4.
> >>
> >> Convert on read has been selected as a good way, because it is not invasive and 
> >> does not limit fresh database. But, this way needs core modification which 
> >> allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 
> >> will be implemented. Sorry about that, but we need move forward and there is not 
> >> easy way without core modification to do it.
> >>
> >> As I mentioned manytimes before there are two major issues with convert on read 
> >> and one small issue.
> >>
> >> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
> >> which reserve space on each page, before upgrade.
> > 
> > Rather than specifying free space as an amount, I was thinking of having
> > a boolean like 'ready_for_upgrade' and the system internally would know
> > how much free space for each page and tuple.
> 
> You need booth, flag which shows you that the relation/database is ready for 
> upgrade and free space reservation configuration for each column. System cannot 
> know it because PostgreSQL is not oracle :-). It does not know what will happend 
> during next version development :-). It have to be setup by pre-upgrade script.

So every Postgres version will allow space reservation based on the
table and page, and the pre-upgrade script will set those values based
on the table's columns --- makes sense, and avoids the need for a minor
update to Postgres to add new code --- good idea.

--  Bruce Momjian  <bruce@momjian.us>        http://momjian.us EnterpriseDB
http://enterprisedb.com
 + If your life is a hard drive, Christ can be your backup. +


Re: pg_upgrade project status

From
Zdenek Kotala
Date:
Bruce Momjian napsal(a):
> Zdenek Kotala wrote:
>> Bruce Momjian napsal(a):
>>> Zdenek Kotala wrote:

<snip>

>>>> 1) Data does not fit on the new page. It will be solve by pre-upgrade check 
>>>> which reserve space on each page, before upgrade.
>>> Rather than specifying free space as an amount, I was thinking of having
>>> a boolean like 'ready_for_upgrade' and the system internally would know
>>> how much free space for each page and tuple.
>> You need booth, flag which shows you that the relation/database is ready for 
>> upgrade and free space reservation configuration for each column. System cannot 
>> know it because PostgreSQL is not oracle :-). It does not know what will happend 
>> during next version development :-). It have to be setup by pre-upgrade script.
> 
> So every Postgres version will allow space reservation based on the
> table and page, and the pre-upgrade script will set those values based
> on the table's columns --- makes sense, and avoids the need for a minor
> update to Postgres to add new code --- good idea.

yes, exactly.
    Zdenek