Thread: Proposal: PageLayout footprint
Current content of control file is insufficient to check if database is
compatible with postgres server. I think we should create footprint file which
will contains all page data structures and their member offset and length (and
maybe more).
As a first step which is useful at least for in-place-upgrade is footprint
switch which prints page layout structures. I attach WIP patch.
I'm thinking also to create pg_footprint view with (version, structure name,
attribute, offset, size) columns. But I'm not sure about usefulness of this
catalog (?maybe for regression tests?).
Comments, ideas?
Thanks Zdenek
--
Zdenek Kotala Sun Microsystems
Prague, Czech Republic http://sun.com/postgresql
*** pgsql_footprint.76863ce445f3/src/backend/main/main.c po srp 11 06:48:20 2008
--- pgsql_footprint/src/backend/main/main.c ne srp 10 19:15:03 2008
***************
*** 34,39 ****
--- 34,40 ----
#include <sys/param.h>
#endif
+ #include "access/itup.h"
#include "bootstrap/bootstrap.h"
#include "postmaster/postmaster.h"
#include "tcop/tcopprot.h"
***************
*** 44,50 ****
#include "libpq/pqsignal.h"
#endif
-
const char *progname;
--- 45,50 ----
***************
*** 52,60 ****
static void help(const char *progname);
static void check_root(const char *progname);
static char *get_current_username(const char *progname);
-
int
main(int argc, char *argv[])
{
--- 52,60 ----
static void help(const char *progname);
static void check_root(const char *progname);
static char *get_current_username(const char *progname);
+ static void footprint(void);
int
main(int argc, char *argv[])
{
***************
*** 149,154 ****
--- 149,159 ----
puts("postgres (PostgreSQL) " PG_VERSION);
exit(0);
}
+ if ( strcmp(argv[1], "--footprint") == 0 )
+ {
+ footprint();
+ exit(0);
+ }
}
/*
***************
*** 301,306 ****
--- 306,312 ----
printf(_(" -t pa|pl|ex show timings after each query\n"));
printf(_(" -T send SIGSTOP to all backend servers if one dies\n"));
printf(_(" -W NUM wait NUM seconds to allow attach from a debugger\n"));
+ printf(_(" --footprint output page structure layout, then exit\n"));
printf(_("\nOptions for single-user mode:\n"));
printf(_(" --single selects single-user mode (must be first argument)\n"));
***************
*** 396,398 ****
--- 402,439 ----
return name;
#endif
}
+
+ #define PRT_STRUCT(str) \
+ printf("%s (%i/%i)\n", CppAsString(str), sizeof(str), MAXALIGN(sizeof(str)));
+ #define PRT_ATTR(str, attr) \
+ printf("\t%s - %i,%i\n", CppAsString(attr), offsetof(str,attr), sizeof(((str *)0)->attr));
+
+ static void
+ footprint(void)
+ {
+ PRT_STRUCT(PageHeaderData);
+ PRT_ATTR(PageHeaderData, pd_lsn);
+ PRT_ATTR(PageHeaderData, pd_tli);
+ PRT_ATTR(PageHeaderData, pd_flags);
+ PRT_ATTR(PageHeaderData, pd_lower);
+ PRT_ATTR(PageHeaderData, pd_upper);
+ PRT_ATTR(PageHeaderData, pd_special);
+ PRT_ATTR(PageHeaderData, pd_pagesize_version);
+ PRT_ATTR(PageHeaderData, pd_prune_xid);
+ PRT_ATTR(PageHeaderData, pd_linp);
+
+ PRT_STRUCT(HeapTupleHeaderData);
+ PRT_ATTR(HeapTupleHeaderData, t_choice.t_heap.t_xmin);
+ PRT_ATTR(HeapTupleHeaderData, t_choice.t_heap.t_xmax);
+ PRT_ATTR(HeapTupleHeaderData, t_choice.t_heap.t_field3.t_cid);
+ PRT_ATTR(HeapTupleHeaderData, t_ctid);
+ PRT_ATTR(HeapTupleHeaderData, t_infomask2);
+ PRT_ATTR(HeapTupleHeaderData, t_infomask);
+ PRT_ATTR(HeapTupleHeaderData, t_hoff);
+ PRT_ATTR(HeapTupleHeaderData, t_bits);
+
+ PRT_STRUCT(IndexTupleData);
+ PRT_ATTR(IndexTupleData, t_tid);
+ PRT_ATTR(IndexTupleData, t_info);
+ }
+
Zdenek Kotala wrote: > Current content of control file is insufficient to check if database is > compatible with postgres server. It is? Do you have an example of where it's insufficient? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a): > Zdenek Kotala wrote: >> Current content of control file is insufficient to check if database >> is compatible with postgres server. > > It is? Do you have an example of where it's insufficient? > Current control file contain following information (related to page layout): maxAlign blcksz toast_max_chunk_size But you don't have control how aligned is each member of data structure. By my opinion -fipa-struct-reorg GCC option could break structure. And maybe there are more compiler magic switches and optimization on different platforms which can modify structure alignment or member order. It probably does not happen often but footprint should protect people to shot himself. Zdenek PS: And of course toast_max_chunk_size is not insufficient as well. There are more constants like MaxHeapTupleSize and so on, but it is different story.
Zdenek Kotala wrote: > By my opinion -fipa-struct-reorg GCC option could break structure. That option would probably break a lot of things. Like our "variable-sized array as last field of a struct" hacks. > And > maybe there are more compiler magic switches and optimization on > different platforms which can modify structure alignment or member > order. It probably does not happen often but footprint should protect > people to shot himself. We depend on a certain member order and alignment rules. If we're worried about that, we should add checks in configure instead, to barf if you try to use such options. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
"Zdenek Kotala" <Zdenek.Kotala@Sun.COM> writes: > By my opinion -fipa-struct-reorg GCC option could break structure. And maybe > there are more compiler magic switches and optimization on different platforms > which can modify structure alignment or member order. It probably does not > happen often but footprint should protect people to shot himself. My version of GCC doesn't have that option, what does it do? If structure members aren't in the order they're defined and padded to the alignment they're declared to have in pg_type then Postgres catalogs won't work anyways. -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's RemoteDBA services!
Gregory Stark wrote: > "Zdenek Kotala" <Zdenek.Kotala@Sun.COM> writes: >> By my opinion -fipa-struct-reorg GCC option could break structure. And maybe >> there are more compiler magic switches and optimization on different platforms >> which can modify structure alignment or member order. It probably does not >> happen often but footprint should protect people to shot himself. > > My version of GCC doesn't have that option, what does it do? From gcc man page: > -fipa-struct-reorg > Perform structure reorganization optimization, that change C-like > structures layout in order to better utilize spatial locality. > This transformation is affective for programs containing arrays of > structures. Available in two compilation modes: profile-based > (enabled with -fprofile-generate) or static (which uses built-in > heuristics). Require -fipa-type-escape to provide the safety of > this transformation. It works only in whole program mode, so it > requires -fwhole-program and -combine to be enabled. Structures > considered cold by this transformation are not affected (see > --param struct-reorg-cold-struct-ratio=value). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> By my opinion -fipa-struct-reorg GCC option could break structure.
>
> That option would probably break a lot of things. Like our
> "variable-sized array as last field of a struct" hacks.
Yes, it is extreme case.
>> And maybe there are more compiler magic switches and optimization on
>> different platforms which can modify structure alignment or member
>> order. It probably does not happen often but footprint should protect
>> people to shot himself.
>
> We depend on a certain member order and alignment rules. If we're
> worried about that, we should add checks in configure instead, to barf
> if you try to use such options.
>
You are able to check order, but you cannot complain about structure member
alignment during configure time. But if you have two binaries which you get from
two sources then you need to verify that both binaries has same structure footprint.
Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but MAXALIGN
is same on SPARC for both binaries, but I'm not sure if it works correctly.
Any other usage is to protect developers to make a mistake and break silently
compatibility, but it should be caught by --footprint switch.
Zdenek
Zdenek Kotala wrote: > Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but > MAXALIGN is same on SPARC for both binaries, but I'm not sure if it > works correctly. Are 32/64-bit binaries on Sparc incompatible, then? Does our control file check catch it? If not, what's causing it, and would the --footprint switch catch it? -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> Similar is 32/64 bit compilation. It is handled on x86 by MAXALIGN but
>> MAXALIGN is same on SPARC for both binaries, but I'm not sure if it
>> works correctly.
>
> Are 32/64-bit binaries on Sparc incompatible, then? Does our control
> file check catch it? If not, what's causing it, and would the
> --footprint switch catch it?
>
It is what I don't know. controlfile only says "incorrect checksum in control
file". It seems to me that CRC computing does not work correctly. I check
pg_control footprint and it is same for 32/64. It seems to me a false positive
complain, but ...
Footprint switch should help show you why it is incompatible and also protect
you during development to break some structure.
Zdenek
Zdenek Kotala wrote: > It is what I don't know. controlfile only says "incorrect checksum in > control file". It seems to me that CRC computing does not work > correctly. I check pg_control footprint and it is same for 32/64. It > seems to me a false positive complain, but ... Endianness perhaps? Per comment in ControlFileData: > * This data is used to check for hardware-architecture compatibility of > * the database and the backend executable. We need not check endianness > * explicitly, since the pg_control version will surely look wrong to a > * machine of different endianness, but we do need to worry about MAXALIGN > * and floating-point format. (Note: storage layout nominally also > * depends on SHORTALIGN and INTALIGN, but in practice these are the same > * on all architectures of interest.) > Footprint switch should help show you why it is incompatible and also > protect you during development to break some structure. Apparently the footprint switch didn't provide any insight into the Sparc 32/64-bit issue, so I'm not too impressed.. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com
Heikki Linnakangas napsal(a):
> Zdenek Kotala wrote:
>> It is what I don't know. controlfile only says "incorrect checksum in
>> control file". It seems to me that CRC computing does not work
>> correctly. I check pg_control footprint and it is same for 32/64. It
>> seems to me a false positive complain, but ...
>
> Endianness perhaps? Per comment in ControlFileData:
No. It is same Endianness. SPARC allows to map memory with different endian, but
it needs extra magic.
Zdenek