Thread: Make documentation builds reproducible
Somewhere at PGCon, I forgot exactly where, maybe in the same meeting where we talked about getting rid of distprep, we talked about that the documentation builds are not reproducible (in the sense of https://reproducible-builds.org/). This is easily fixable, the fix is available upstream (https://github.com/docbook/xslt10-stylesheets/issues/54) but not released. We can backpatch that into our customization layer. The attached patch shows it. I had actually often wanted this during development. When making documentation tooling changes, it's useful to be able to compare the output before and after, and this will eliminate false positives in that. This patch addresses both the HTML and the FO output. The man output is already reproducible AFAICT. Note that the final PDF output is currently not reproducible; that's a different issue that needs to be fixed in FOP. (See https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.)
Attachment
On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: > Somewhere at PGCon, I forgot exactly where, maybe in the same meeting > where we talked about getting rid of distprep, we talked about that the > documentation builds are not reproducible (in the sense of > https://reproducible-builds.org/). This is easily fixable, the fix is > available upstream > (https://github.com/docbook/xslt10-stylesheets/issues/54) but not > released. We can backpatch that into our customization layer. The > attached patch shows it. I am a tiny bit confused here. The commit that solved the issue was merged into the master branch in 2018. GitHub lists the lastest release as being in 2020. A quick git command shows this has been in releases since December of 2018. $ git --no-pager tag --contains 0763160 ndw-test-001 snapshot-2018-12-07-01 snapshot-ndw-test/2019-10-04 snapshot/2018-09-28-172 snapshot/2018-09-28-173 snapshot/2018-09-28-174 snapshot/2018-09-28-175 snapshot/2018-09-29-176 snapshot/2018-09-29-177 snapshot/2018-09-30-178 snapshot/2018-09-30-179 snapshot/2018-10-01-180 snapshot/2018-10-02-183 snapshot/2018-10-02-184 snapshot/2018-10-16-185 snapshot/2018-10-16-186 snapshot/2018-10-21-188 snapshot/2018-11-01-191 snapshot/2019-10-05-bobs snapshot/2020-05-28-pdesjardins snapshot/2020-06-03 Is there anything I am missing? Is Postgres relying on releases older than snapshot-2018-12-07-01? If so, is it possible to up the minimum version? > I had actually often wanted this during development. When making > documentation tooling changes, it's useful to be able to compare the > output before and after, and this will eliminate false positives in that. > > This patch addresses both the HTML and the FO output. The man output is > already reproducible AFAICT. Note that the final PDF output is > currently not reproducible; that's a different issue that needs to be > fixed in FOP. (See > https://wiki.debian.org/ReproducibleBuilds/TimestampsInPDFGeneratedByApacheFOP.) I think reproducibility is very important. Thanks for taking this on! -- Tristan Partin Neon (https://neon.tech)
"Tristan Partin" <tristan@neon.tech> writes: > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting >> where we talked about getting rid of distprep, we talked about that the >> documentation builds are not reproducible (in the sense of >> https://reproducible-builds.org/). This is easily fixable, > Is there anything I am missing? Is Postgres relying on releases older > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > version? AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, which seems to have been released in 2017, so it's unsurprising that it's missing this fix. It's kind of hard to argue that developers (much less distro packagers) should install unsupported snapshot releases in order to build our docs. Having said that, maybe we should check whether this patch is compatible with those snapshot releases, just in case somebody is using one. regards, tom lane
On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: > "Tristan Partin" <tristan@neon.tech> writes: > > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: > >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting > >> where we talked about getting rid of distprep, we talked about that the > >> documentation builds are not reproducible (in the sense of > >> https://reproducible-builds.org/). This is easily fixable, > > > Is there anything I am missing? Is Postgres relying on releases older > > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > > version? > > AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, > which seems to have been released in 2017, so it's unsurprising that > it's missing this fix. > > It's kind of hard to argue that developers (much less distro packagers) > should install unsupported snapshot releases in order to build our docs. > Having said that, maybe we should check whether this patch is compatible > with those snapshot releases, just in case somebody is using one. I agree with you. Thanks for the pointer. -- Tristan Partin Neon (https://neon.tech)
On Fri, 25 Aug 2023 at 01:23, Tristan Partin <tristan@neon.tech> wrote: > > On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: > > "Tristan Partin" <tristan@neon.tech> writes: > > > On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: > > >> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting > > >> where we talked about getting rid of distprep, we talked about that the > > >> documentation builds are not reproducible (in the sense of > > >> https://reproducible-builds.org/). This is easily fixable, > > > > > Is there anything I am missing? Is Postgres relying on releases older > > > than snapshot-2018-12-07-01? If so, is it possible to up the minimum > > > version? > > > > AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, > > which seems to have been released in 2017, so it's unsurprising that > > it's missing this fix. > > > > It's kind of hard to argue that developers (much less distro packagers) > > should install unsupported snapshot releases in order to build our docs. > > Having said that, maybe we should check whether this patch is compatible > > with those snapshot releases, just in case somebody is using one. > > I agree with you. Thanks for the pointer. I'm seeing that there has been no activity in this thread for nearly 5 months, I'm planning to close this in the current commitfest unless someone is planning to take it forward. Regards, Vignesh
On 20.01.24 03:33, vignesh C wrote: > On Fri, 25 Aug 2023 at 01:23, Tristan Partin <tristan@neon.tech> wrote: >> >> On Thu Aug 24, 2023 at 2:30 PM CDT, Tom Lane wrote: >>> "Tristan Partin" <tristan@neon.tech> writes: >>>> On Wed Aug 23, 2023 at 2:24 PM CDT, Peter Eisentraut wrote: >>>>> Somewhere at PGCon, I forgot exactly where, maybe in the same meeting >>>>> where we talked about getting rid of distprep, we talked about that the >>>>> documentation builds are not reproducible (in the sense of >>>>> https://reproducible-builds.org/). This is easily fixable, >>> >>>> Is there anything I am missing? Is Postgres relying on releases older >>>> than snapshot-2018-12-07-01? If so, is it possible to up the minimum >>>> version? >>> >>> AFAICT the "latest stable release" of docbook-xsl is still 1.79.2, >>> which seems to have been released in 2017, so it's unsurprising that >>> it's missing this fix. >>> >>> It's kind of hard to argue that developers (much less distro packagers) >>> should install unsupported snapshot releases in order to build our docs. >>> Having said that, maybe we should check whether this patch is compatible >>> with those snapshot releases, just in case somebody is using one. >> >> I agree with you. Thanks for the pointer. > > I'm seeing that there has been no activity in this thread for nearly 5 > months, I'm planning to close this in the current commitfest unless > someone is planning to take it forward. I think there was general agreement with what this patch is doing, but I guess it's too boring to actually review the patch in detail. Let's say, if there are no objections, I'll go ahead and commit it.
Peter Eisentraut <peter@eisentraut.org> writes:
> I think there was general agreement with what this patch is doing, but I
> guess it's too boring to actually review the patch in detail. Let's
> say, if there are no objections, I'll go ahead and commit it.
I re-read the thread and have two thoughts:
* We worried about whether this change would be compatible with a
(presently unreleased) version of docbook that contains the upstreamed
fix. It seems unlikely that there's a problem, but maybe worth
checking?
* I gather that the point here is to change some generated anchor
tags. Would any of these tags be things people would be likely
to have bookmarked?
regards, tom lane
On 20.01.24 17:03, Tom Lane wrote: > Peter Eisentraut <peter@eisentraut.org> writes: >> I think there was general agreement with what this patch is doing, but I >> guess it's too boring to actually review the patch in detail. Let's >> say, if there are no objections, I'll go ahead and commit it. > > I re-read the thread and have two thoughts: > > * We worried about whether this change would be compatible with a > (presently unreleased) version of docbook that contains the upstreamed > fix. It seems unlikely that there's a problem, but maybe worth > checking? The code in the patch is the same code as upstream, so it would behave the same as a new release. > * I gather that the point here is to change some generated anchor > tags. Would any of these tags be things people would be likely > to have bookmarked? No, because the problem is that the anchor names are randomly generated in each build.
Peter Eisentraut <peter@eisentraut.org> writes:
> On 20.01.24 17:03, Tom Lane wrote:
>> * I gather that the point here is to change some generated anchor
>> tags. Would any of these tags be things people would be likely
>> to have bookmarked?
> No, because the problem is that the anchor names are randomly generated
> in each build.
D'oh. No objection then.
regards, tom lane
On 20.01.24 23:59, Tom Lane wrote: > Peter Eisentraut <peter@eisentraut.org> writes: >> On 20.01.24 17:03, Tom Lane wrote: >>> * I gather that the point here is to change some generated anchor >>> tags. Would any of these tags be things people would be likely >>> to have bookmarked? > >> No, because the problem is that the anchor names are randomly generated >> in each build. > > D'oh. No objection then. Thanks, committed.
Hi,
I usually the HTML documentation locally using command:
make STYLE=website html
~
This has been working forever, but seems to have broken due to commit
[1] having an undeclared variable.
e.g.
[postgres@CentOS7-x64 sgml]$ make STYLE=website html
{ \
echo "<!ENTITY version \"17devel\">"; \
echo "<!ENTITY majorversion \"17\">"; \
} > version.sgml
'/usr/bin/perl' ./mk_feature_tables.pl YES
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-supported.sgml
'/usr/bin/perl' ./mk_feature_tables.pl NO
../../../src/backend/catalog/sql_feature_packages.txt
../../../src/backend/catalog/sql_features.txt >
features-unsupported.sgml
'/usr/bin/perl' ./generate-errcodes-table.pl
../../../src/backend/utils/errcodes.txt > errcodes-table.sgml
'/usr/bin/perl' ./generate-keywords-table.pl . > keywords-table.sgml
'/usr/bin/perl' ./generate-targets-meson.pl targets-meson.txt
generate-targets-meson.pl > targets-meson.sgml
'/usr/bin/perl'
../../../src/backend/utils/activity/generate-wait_event_types.pl
--docs ../../../src/backend/utils/activity/wait_event_names.txt
/usr/bin/xmllint --nonet --path . --path . --output postgres-full.xml
--noent --valid postgres.sgml
/usr/bin/xsltproc --nonet --path . --path . --stringparam pg.version
'17devel' --param website.stylesheet 1 stylesheet.xsl
postgres-full.xml
runtime error: file stylesheet-html-common.xsl line 452 element if
Variable 'autolink.index.see' has not been declared.
make: *** [html-stamp] Error 10
======
[1] https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1
Kind Regards,
Peter Smith.
Fujitsu Australia
Peter Smith <smithpb2250@gmail.com> writes:
> I usually the HTML documentation locally using command:
> make STYLE=website html
> This has been working forever, but seems to have broken due to commit
> [1] having an undeclared variable.
Interestingly, that still works fine for me, on RHEL8 with
docbook-dtds-1.0-69.el8.noarch
docbook-style-xsl-1.79.2-9.el8.noarch
docbook-style-dsssl-1.79-25.el8.noarch
What docbook version are you using?
regards, tom lane
On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > Peter Smith <smithpb2250@gmail.com> writes: > > I usually the HTML documentation locally using command: > > make STYLE=website html > > This has been working forever, but seems to have broken due to commit > > [1] having an undeclared variable. > > Interestingly, that still works fine for me, on RHEL8 with > > docbook-dtds-1.0-69.el8.noarch > docbook-style-xsl-1.79.2-9.el8.noarch > docbook-style-dsssl-1.79-25.el8.noarch > > What docbook version are you using? > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook docbook-dtds.noarch 1.0-60.el7 @anaconda docbook-style-dsssl.noarch 1.79-18.el7 @base docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda ====== Kind Regards, Peter Smith. Fujitsu Australia
On Tue, Jan 23, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote: > > On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > > > Peter Smith <smithpb2250@gmail.com> writes: > > > I usually the HTML documentation locally using command: > > > make STYLE=website html > > > This has been working forever, but seems to have broken due to commit > > > [1] having an undeclared variable. > > > > Interestingly, that still works fine for me, on RHEL8 with > > > > docbook-dtds-1.0-69.el8.noarch > > docbook-style-xsl-1.79.2-9.el8.noarch > > docbook-style-dsssl-1.79-25.el8.noarch > > > > What docbook version are you using? > > > > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook > docbook-dtds.noarch 1.0-60.el7 @anaconda > docbook-style-dsssl.noarch 1.79-18.el7 @base > docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda > IIUC these releases notes [1] say autolink.index.see existed since v1.79.1, but unfortunately, that is more recent than my ancient installed v1.78.1 From the release notes: ------ Robert Stayton: autolink.index.see.xml New param to control automatic links in index from see and seealso to indexterm primary. ------ ====== [1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html Kind Regards, Peter Smith. Fujitsu Australia
On Thu, Jan 25, 2024 at 9:12 AM Peter Smith <smithpb2250@gmail.com> wrote: > > On Tue, Jan 23, 2024 at 12:32 PM Peter Smith <smithpb2250@gmail.com> wrote: > > > > On Tue, Jan 23, 2024 at 12:13 PM Tom Lane <tgl@sss.pgh.pa.us> wrote: > > > > > > Peter Smith <smithpb2250@gmail.com> writes: > > > > I usually the HTML documentation locally using command: > > > > make STYLE=website html > > > > This has been working forever, but seems to have broken due to commit > > > > [1] having an undeclared variable. > > > > > > Interestingly, that still works fine for me, on RHEL8 with > > > > > > docbook-dtds-1.0-69.el8.noarch > > > docbook-style-xsl-1.79.2-9.el8.noarch > > > docbook-style-dsssl-1.79-25.el8.noarch > > > > > > What docbook version are you using? > > > > > > > [postgres@CentOS7-x64 sgml]$ sudo yum list installed | grep docbook > > docbook-dtds.noarch 1.0-60.el7 @anaconda > > docbook-style-dsssl.noarch 1.79-18.el7 @base > > docbook-style-xsl.noarch 1.78.1-3.el7 @anaconda > > > > IIUC these releases notes [1] say autolink.index.see existed since > v1.79.1, but unfortunately, that is more recent than my ancient > installed v1.78.1 > > From the release notes: > ------ > Robert Stayton: autolink.index.see.xml > > New param to control automatic links in index from see and > seealso to indexterm primary. > ------ > > ====== > [1] https://docbook.sourceforge.net/release/xsl/1.79.1/RELEASE-NOTES.html > Is anything going to be changed for this? Since the recent commit [1] when building the docs now each time I need to first hack (e.g. either the Makefile or stylesheet-html-common.xml) to declare the missing ‘autolink.index.see’ variable. I know that my old OS is approaching EOL but I thought my docbook installation was still valid. ====== [1] https://github.com/postgres/postgres/commit/b0f0a9432d0b6f53634a96715f2666f6d4ea25a1 Kind Regards, Peter Smith. Fujitsu Australia
Peter Smith <smithpb2250@gmail.com> writes:
>> IIUC these releases notes [1] say autolink.index.see existed since
>> v1.79.1, but unfortunately, that is more recent than my ancient
>> installed v1.78.1
> Is anything going to be changed for this?
I assume Peter E. is going to address it, but FOSDEM is this week and
so a lot of people are going to be busy traveling and conferencing
rather than hacking. Things might not happen right away.
You could possibly help move things along if you can propose a
workable patch.
regards, tom lane
On 23.01.24 02:06, Peter Smith wrote: > This has been working forever, but seems to have broken due to commit > [1] having an undeclared variable. > runtime error: file stylesheet-html-common.xsl line 452 element if > Variable 'autolink.index.see' has not been declared. > make: *** [html-stamp] Error 10 I have committed a fix for this. I have successfully tested docbook-xsl 1.77.1 through 1.79.*.
On Thu, Feb 8, 2024 at 9:47 PM Peter Eisentraut <peter@eisentraut.org> wrote: > > On 23.01.24 02:06, Peter Smith wrote: > > This has been working forever, but seems to have broken due to commit > > [1] having an undeclared variable. > > > runtime error: file stylesheet-html-common.xsl line 452 element if > > Variable 'autolink.index.see' has not been declared. > > make: *** [html-stamp] Error 10 > > I have committed a fix for this. I have successfully tested docbook-xsl > 1.77.1 through 1.79.*. > Yes, the latest is working for me now. Thanks! ====== Kind Regards, Peter Smith. Fujitsu Australia.