Thread: pg_upgrade project status
In the last week community made decision about pg_upgrade project and its
implementation. I would like to try summarize this conclusion and I add other
topic which should be finished for 8.4.
Convert on read has been selected as a good way, because it is not invasive and
does not limit fresh database. But, this way needs core modification which
allows to do online in-place upgrade. It means no online in-place upgrade to 8.4
will be implemented. Sorry about that, but we need move forward and there is not
easy way without core modification to do it.
As I mentioned manytimes before there are two major issues with convert on read
and one small issue.
1) Data does not fit on the new page. It will be solve by pre-upgrade check
which reserve space on each page, before upgrade.
2) No information about TOASTed data in TOAST table. It is problem for example
when varlena encoding is changed, or heaptupleheader or array data stucture.
Idea is to add column position into each chunk and also replace chunk ID by data
offset.
3) Access methods stores "shadow" data into pages - for example metapages or
hash index has bitmap pages, but there is no information on the page about this
and page conversion could lost the data. We need mark this pages. Metapage is
usually first page and it could be hardwired, but hash bitmap can be everywhere.
I suggest to add page flag HAS_EXTRA_CONTENT (give me a better name).
IIRC, somebody suggest to replace HASH bitmap pages with FSM? Any idea?
I hope that I mention all. Please feel free to add more information or correct
me if I wrong.
Thanks Zdenek
Zdenek Kotala wrote: > In the last week community made decision about pg_upgrade project and its > implementation. I would like to try summarize this conclusion and I add other > topic which should be finished for 8.4. > > Convert on read has been selected as a good way, because it is not invasive and > does not limit fresh database. But, this way needs core modification which > allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 > will be implemented. Sorry about that, but we need move forward and there is not > easy way without core modification to do it. > > As I mentioned manytimes before there are two major issues with convert on read > and one small issue. > > 1) Data does not fit on the new page. It will be solve by pre-upgrade check > which reserve space on each page, before upgrade. Rather than specifying free space as an amount, I was thinking of having a boolean like 'ready_for_upgrade' and the system internally would know how much free space for each page and tuple. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian <bruce@momjian.us> writes: >> 1) Data does not fit on the new page. It will be solve by pre-upgrade check >> which reserve space on each page, before upgrade. > > Rather than specifying free space as an amount, I was thinking of having > a boolean like 'ready_for_upgrade' and the system internally would know > how much free space for each page and tuple. I think the original idea was that 8.5 would come with a tool which connected to an 8.4 database and makes sure things are ok. To do so it has to do two things: 1) set some server state so that the server doesn't create any new "bad" pages and 2) check that all the existing pages and fix any "bad" pages. The question here is what state does the server need to have to ensure it doesn't create any new "bad" pages. The simplest option would be a "minimum_free_space_per_page" guc variable. That would be sufficient if we're worried about expanding the page header. The user wouldn't have to know about these, the tool would set it for him. If we're worried about expanding tuple header overhead then we would need a separate option. If we grow any data type representations then we could still have a problem. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's 24x7 Postgres support!
Gregory Stark wrote: > Bruce Momjian <bruce@momjian.us> writes: > > >> 1) Data does not fit on the new page. It will be solve by pre-upgrade check > >> which reserve space on each page, before upgrade. > > > > Rather than specifying free space as an amount, I was thinking of having > > a boolean like 'ready_for_upgrade' and the system internally would know > > how much free space for each page and tuple. > > I think the original idea was that 8.5 would come with a tool which connected > to an 8.4 database and makes sure things are ok. To do so it has to do two > things: 1) set some server state so that the server doesn't create any new > "bad" pages and 2) check that all the existing pages and fix any "bad" pages. > > The question here is what state does the server need to have to ensure it > doesn't create any new "bad" pages. The simplest option would be a > "minimum_free_space_per_page" guc variable. That would be sufficient if we're > worried about expanding the page header. The user wouldn't have to know about > these, the tool would set it for him. > > If we're worried about expanding tuple header overhead then we would need a > separate option. If we grow any data type representations then we could still > have a problem. Yes, this is why I was thinking it should just be a boolean and the old server will know the requirements, but it does require us to force a minor upgrade on the old server so it has the proper information embedded in the binary. We could force each page to have the required amount of free space but the binary is going to need to not invalidate that as database processing continues before the upgrade-in-place. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian napsal(a):
> Zdenek Kotala wrote:
>> In the last week community made decision about pg_upgrade project and its
>> implementation. I would like to try summarize this conclusion and I add other
>> topic which should be finished for 8.4.
>>
>> Convert on read has been selected as a good way, because it is not invasive and
>> does not limit fresh database. But, this way needs core modification which
>> allows to do online in-place upgrade. It means no online in-place upgrade to 8.4
>> will be implemented. Sorry about that, but we need move forward and there is not
>> easy way without core modification to do it.
>>
>> As I mentioned manytimes before there are two major issues with convert on read
>> and one small issue.
>>
>> 1) Data does not fit on the new page. It will be solve by pre-upgrade check
>> which reserve space on each page, before upgrade.
>
> Rather than specifying free space as an amount, I was thinking of having
> a boolean like 'ready_for_upgrade' and the system internally would know
> how much free space for each page and tuple.
You need booth, flag which shows you that the relation/database is ready for
upgrade and free space reservation configuration for each column. System cannot
know it because PostgreSQL is not oracle :-). It does not know what will happend
during next version development :-). It have to be setup by pre-upgrade script.
Zdenek
Zdenek Kotala wrote: > Bruce Momjian napsal(a): > > Zdenek Kotala wrote: > >> In the last week community made decision about pg_upgrade project and its > >> implementation. I would like to try summarize this conclusion and I add other > >> topic which should be finished for 8.4. > >> > >> Convert on read has been selected as a good way, because it is not invasive and > >> does not limit fresh database. But, this way needs core modification which > >> allows to do online in-place upgrade. It means no online in-place upgrade to 8.4 > >> will be implemented. Sorry about that, but we need move forward and there is not > >> easy way without core modification to do it. > >> > >> As I mentioned manytimes before there are two major issues with convert on read > >> and one small issue. > >> > >> 1) Data does not fit on the new page. It will be solve by pre-upgrade check > >> which reserve space on each page, before upgrade. > > > > Rather than specifying free space as an amount, I was thinking of having > > a boolean like 'ready_for_upgrade' and the system internally would know > > how much free space for each page and tuple. > > You need booth, flag which shows you that the relation/database is ready for > upgrade and free space reservation configuration for each column. System cannot > know it because PostgreSQL is not oracle :-). It does not know what will happend > during next version development :-). It have to be setup by pre-upgrade script. So every Postgres version will allow space reservation based on the table and page, and the pre-upgrade script will set those values based on the table's columns --- makes sense, and avoids the need for a minor update to Postgres to add new code --- good idea. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian napsal(a):
> Zdenek Kotala wrote:
>> Bruce Momjian napsal(a):
>>> Zdenek Kotala wrote:
<snip>
>>>> 1) Data does not fit on the new page. It will be solve by pre-upgrade check
>>>> which reserve space on each page, before upgrade.
>>> Rather than specifying free space as an amount, I was thinking of having
>>> a boolean like 'ready_for_upgrade' and the system internally would know
>>> how much free space for each page and tuple.
>> You need booth, flag which shows you that the relation/database is ready for
>> upgrade and free space reservation configuration for each column. System cannot
>> know it because PostgreSQL is not oracle :-). It does not know what will happend
>> during next version development :-). It have to be setup by pre-upgrade script.
>
> So every Postgres version will allow space reservation based on the
> table and page, and the pre-upgrade script will set those values based
> on the table's columns --- makes sense, and avoids the need for a minor
> update to Postgres to add new code --- good idea.
yes, exactly.
Zdenek