Thread: Proposal: manipulating pg_control file from Perl
Hi, Previously it was pointed out [1] that manipulating the pg_control file from Cluster.pm may be beneficial under certain conditions. As an example, one can downgrade the catalog version in pg_control, place input files for pg_upgrade, execute pg_upgrade and check the results. This would allow writing TAP-tests for pg_upgrade easily, in some cases at least. Perhaps there are other scenarios. Unfortunately this can't be done at the moment. One of the reasons is that the ControlFileData structure stores platform-dependent types like `int`. The size of the structure and all the offsets within it are variable. The second reason is that integer representation may be little- or big-endian depending on the platform. The third reason is that the alignment of the fields may differ even if we switch to types of the known size like int32 / int64. For named reasons, manipulating pg_upgrade from Perl is impractical, considering the number of environments we support. I see two possible solutions: 1) Provide a tool written in C that allows changing pg_control, e.g. `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. The tool can be executed from Perl, so it shouldn't know about sizeof(int), alignment and endiness. 2) Change the format of pg_control to platform-independent, semi-textual perhaps. The reason why it should be semi- is that we should store the checksum of the file and I think it should have a fixed offset. For instance: ``` # This is auto-generated file, don't edit! checksum=AABBCCDD pg_conrol_version=1800 cat_version=202503131 ... etc ... ``` In case (2) we may consider getting rid of the pg_controldata tool. Thoughts? [1]: https://www.postgresql.org/message-id/ZzVOTc3ZgPWfEQut%40paquier.xyz -- Best regards, Aleksander Alekseev
Aleksander Alekseev <aleksander@timescale.com> writes:
> Previously it was pointed out [1] that manipulating the pg_control
> file from Cluster.pm may be beneficial under certain conditions.
> As an example, one can downgrade the catalog version in pg_control,
> place input files for pg_upgrade, execute pg_upgrade and check the
> results. This would allow writing TAP-tests for pg_upgrade easily, in
> some cases at least. Perhaps there are other scenarios.
I'm finding it *really* hard to buy that as a credible use-case.
There is no situation where changing the catalog version in
pg_control, by itself, is a useful thing to do: you'd have to also
install matching catalog contents. But then wherever you got the
catalog contents from would be a perfectly good source for a matching
pg_control, no? Likewise, the idea that pg_control should be
architecture-independent when none of the catalogs or data files are
seems pretty pointless.
Maybe there's a case to be made for doing something here, but you
haven't made it.
regards, tom lane
Aleksander Alekseev <aleksander@timescale.com> writes: > Hi, > > Previously it was pointed out [1] that manipulating the pg_control > file from Cluster.pm may be beneficial under certain conditions. [...] > Unfortunately this can't be done at the moment. One of the reasons is > that the ControlFileData structure stores platform-dependent types > like `int`. The size of the structure and all the offsets within it > are variable. The second reason is that integer representation may be > little- or big-endian depending on the platform. The third reason is > that the alignment of the fields may differ even if we switch to types > of the known size like int32 / int64. Notwitstanding Tom's objections, these are not reasons for not being able to manipulate these values from Perl. The `i` and `I` formats for pack/unpack (see https://perldoc.perl.org/functions/pack) use what the C compiler calls `int`, in terms of both endianness and size. > For named reasons, manipulating pg_upgrade from Perl is impractical, > considering the number of environments we support. > > I see two possible solutions: > > 1) Provide a tool written in C that allows changing pg_control, e.g. > `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. The > tool can be executed from Perl, so it shouldn't know about > sizeof(int), alignment and endiness. 1.5) Use Perl's pack/unpack functions, which are explicitly desgined for exactly this use case. - ilmari
Hi Dagfinn, > Notwitstanding Tom's objections, these are not reasons for not being > able to manipulate these values from Perl. The `i` and `I` formats for > pack/unpack (see https://perldoc.perl.org/functions/pack) use what the C > compiler calls `int`, in terms of both endianness and size. > > > For named reasons, manipulating pg_upgrade from Perl is impractical, > > considering the number of environments we support. > > > > I see two possible solutions: > > > > 1) Provide a tool written in C that allows changing pg_control, e.g. > > `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. The > > tool can be executed from Perl, so it shouldn't know about > > sizeof(int), alignment and endiness. > > 1.5) Use Perl's pack/unpack functions, which are explicitly desgined for > exactly this use case. Thanks for your reply. It is my understanding that Perl is not extremely aware of alignment. For instance if I want to modify the checksum of the file and the offset of the checksum is let's say 200 bytes on one platform, 204 on another and 208 on a third, pack/unpack will not help me. Or did I miss something? -- Best regards, Aleksander Alekseev
Re: Aleksander Alekseev > 1) Provide a tool written in C that allows changing pg_control, e.g. > `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. I thought we already had pg_resetwal for that. Christoph
Hi Christoph, > Re: Aleksander Alekseev > > 1) Provide a tool written in C that allows changing pg_control, e.g. > > `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. > > I thought we already had pg_resetwal for that. That's a good point. In the case I referred to [1] however I wanted to manipulate more fields than pg_resetwal currently allows, e.g. catalog version - which, as Tom pointed out, is a controversial thing to do in a TAP test. Do you think pg_resetwall should be able to do more than it currently does, or merely pointing out that we already have a tool for editing pg_control file? 73s de Alex, R2AUK [1]: https://postgr.es/m/CAJ7c6TOy7fUW9MuNeOWor3cSFnQg9tgz%3DmjXHDb94GORtM_Eyg%40mail.gmail.com
Re: Aleksander Alekseev > Do you think pg_resetwall should be able to do more than it currently > does, or merely pointing out that we already have a tool for editing > pg_control file? I don't know if being able to mess with these fields makes sense, but if so, pg_resetwal seems like the correct place to put it. > 73s de Alex, R2AUK 73 TU :) Christoph
Aleksander Alekseev <aleksander@timescale.com> writes: > Hi Dagfinn, > >> Notwitstanding Tom's objections, these are not reasons for not being >> able to manipulate these values from Perl. The `i` and `I` formats for >> pack/unpack (see https://perldoc.perl.org/functions/pack) use what the C >> compiler calls `int`, in terms of both endianness and size. >> >> > For named reasons, manipulating pg_upgrade from Perl is impractical, >> > considering the number of environments we support. >> > >> > I see two possible solutions: >> > >> > 1) Provide a tool written in C that allows changing pg_control, e.g. >> > `pg_writecontoldata` or maybe a flat like `pg_controldata -w`. The >> > tool can be executed from Perl, so it shouldn't know about >> > sizeof(int), alignment and endiness. >> >> 1.5) Use Perl's pack/unpack functions, which are explicitly desgined for >> exactly this use case. > > Thanks for your reply. > > It is my understanding that Perl is not extremely aware of alignment. > For instance if I want to modify the checksum of the file and the > offset of the checksum is let's say 200 bytes on one platform, 204 on > another and 208 on a third, pack/unpack will not help me. > > Or did I miss something? You have to specify alignment manually using the `x![$type]` notation. I actually wrote up a pack template for the entire control file (assuming all types have alignof==sizeof), and it seems to work. Run the attached with a Postgres 17 data directory as the arguemnt, and it should show all the fields. - ilmari
Attachment
Hi, > > It is my understanding that Perl is not extremely aware of alignment. > > For instance if I want to modify the checksum of the file and the > > offset of the checksum is let's say 200 bytes on one platform, 204 on > > another and 208 on a third, pack/unpack will not help me. > > > > Or did I miss something? > > You have to specify alignment manually using the `x![$type]` notation. > I actually wrote up a pack template for the entire control file > (assuming all types have alignof==sizeof), and it seems to work. Run > the attached with a Postgres 17 data directory as the arguemnt, and it > should show all the fields. Oh that's neat. Thanks for sharing! -- Best regards, Aleksander Alekseev