Aside from the pg_upgrade discussion, here's an updated version of the patch, rebased over master. It also contains some other misc refactoring I've done while reading through the patch. I haven't tested this much, I may well have also broken something, but I wanted to post an update before the weekend.
Thinking about the page format, I think we should start using the pd_lower/upper pointers in the data page format. For a non-leaf page, pd_upper would always point to the beginning of the special area, and pd_lower would indicate the end of PostingItems. For a leaf page, pd_lower would indicate the end of the compressed posting list, and pd_upper would point to the "leaf-index" at the end of the page. That matches the standard page layout in the sense that the space between pd_lower and pd_upper is free, although the data stored in the non-free areas would be quite different. That would allow us to mark full-page images with buffer_std, allowing the "gap" to be left out. I think that would be a more natural way to keep track of the used/unused space on the page, anyway, compared to the current maxoff/endoffset field in the special area.
In the attached patch, I in fact already did that for data leaf pages, but didn't change the format of non-leaf pages yet. If we want to support pg_upgrade, we might want to refrain from changing the non-leaf format.
In GinDataLeafPageGetPostingList* you use sizeof(ItemPointerData) without MAXALIGN. Is it an error or you especially use 2 extra bytes on leaf page?
