Thread: meson documentation build open issues
I have identified several open issues with the documentation build under Meson (approximately in priority order): 1. Image files are not handled at all, so they don't show up in the final product. 2. Defaults to website stylesheet, no way to configure. This should be adjusted to match the make build. 3. The various build targets and their combinations are mismatching and incomplete. For example: Top-level GNUmakefile has these targets: - docs (builds html and man) - html - man (Those are the formats that are part of a distribution build.) doc/src/sgml/Makefile has these documented targets: - default target is html - all (builds html and man, maps to top-level "docs") - html - man - postgres-A4.pdf - postgres-US.pdf - check as well as (undocumented): - htmlhelp - postgres.html - postgres.txt - epub - postgres.epub - postgres.info meson has the following documented targets: - docs (builds only html) - alldocs (builds all formats, including obscure ones) as well as the following undocumented targets: - html - man - html_help [sic] - postgres-A4.pdf - postgres-US.pdf - postgres.epub - [info is not implemented at all] - [didn't find an equivalent of check] As you can see, this is all over the place. I'd like to arrive at some consistency across all build systems for handling each tier of documentation formats, in terms of what is documented, what the targets are named, and how they are grouped. 4. There doesn't appear to be a way to install the documentation. (There are also some open questions in the top-level meson.build about the installation directories, but I suppose if we can't install them then exactly where to install them hasn't been thought about too much.) 5. There doesn't appear to be an equivalent of "make world" and "make install-world" that includes documentation builds.
Hi, On 2023-03-15 08:14:09 +0100, Peter Eisentraut wrote: > I have identified several open issues with the documentation build under > Meson (approximately in priority order): > > 1. Image files are not handled at all, so they don't show up in the final > product. Hm. Somehow I thought I'd tackled that at some point. Ah. I got there for the PDF output, but didn't realize it's also an issue for the html output. For FO it sufficed to set the img.src.path param. For HTML that's not enough, because that just adjusts the link to the file - but we don't want to link to the source file. We actually solved this for the single-page html version - we just embed the svg. I wonder if we should just do that as well. Another way would be to emit the files into the desired place as part of the stylesheet. While it requires touching xslt, it does seems somewhat more elegant than just copying files around. I did implement that, curious what you think. > 2. Defaults to website stylesheet, no way to configure. This should be > adjusted to match the make build. Should we add a meson option? > 3. The various build targets and their combinations are mismatching and > incomplete. For example: > > Top-level GNUmakefile has these targets: > > - docs (builds html and man) > - html > - man > > (Those are the formats that are part of a distribution build.) > > doc/src/sgml/Makefile has these documented targets: > > - default target is html > - all (builds html and man, maps to top-level "docs") > - html > - man > - postgres-A4.pdf > - postgres-US.pdf > - check > > as well as (undocumented): > > - htmlhelp > - postgres.html > - postgres.txt > - epub > - postgres.epub > - postgres.info > > meson has the following documented targets: > > - docs (builds only html) > - alldocs (builds all formats, including obscure ones) > > as well as the following undocumented targets: > > - html > - man > - html_help [sic] renamed in the attached patch. > - postgres-A4.pdf > - postgres-US.pdf > - postgres.epub Note that these are actually named doc/src/sgml/{html,man,...}, not top-level targets. > - [info is not implemented at all] Would be easy to implement, but not sure it's worth doing. > - [didn't find an equivalent of check] That's probably worth doing - should it be run as an actual test, or be a target? > 4. There doesn't appear to be a way to install the documentation. > (There are also some open questions in the top-level meson.build about > the installation directories, but I suppose if we can't install them > then exactly where to install them hasn't been thought about too > much.) WIP patch for that attached. There's now install-doc-man install-doc-html run targets and a install-docs alias target. I did end up getting stuck when hacking on this, and ended up adding css support for nochunk and support for the website style for htmlhelp and nochunk, as well as obsoleting the need for copying the css files... But perhaps that's a bit too much. Greetings, Andres Freund
Attachment
Hi, On 2023-03-15 20:55:33 -0700, Andres Freund wrote: > WIP patch for that attached. There's now > install-doc-man > install-doc-html > run targets and a > install-docs > alias target. > > > I did end up getting stuck when hacking on this, and ended up adding css > support for nochunk and support for the website style for htmlhelp and > nochunk, as well as obsoleting the need for copying the css files... But > perhaps that's a bit too much. Updated set of patches attached. This one works in older meson versions too and adds install-world and install-quiet targets. I also ended up getting so frustrated at the docs build speed that I started to hack a bit on that. I attached a patch shaving a few seconds off the buildtime. I think we can make the docs build in parallel and incrementally, by building the different parts of the docs in parallel, using --stringparam rootid, e.g. building each 'part' separately. A very very rough draft attached: parallel with parts: real 0m10.831s user 0m58.295s sys 0m1.402s normal: real 0m32.215s user 0m31.876s sys 0m0.328s 1/3 of the build time at 2x the cost is nothing to sneeze at. Greetings, Andres Freund
Attachment
- v2-0001-meson-rename-html_help-target-to-htmlhelp.patch
- v2-0002-meson-make-install_test_files-more-generic-rename.patch
- v2-0003-wip-meson-add-install-docs-doc-html-doc-man-targe.patch
- v2-0004-meson-add-install-quiet-world-targets.patch
- v2-0005-docs-html-copy-images-to-output-as-part-of-xslt-b.patch
- v2-0006-wip-docs-copy-or-inline-css.patch
- v2-0007-docs-speed-up-docs-build-by-special-casing-the-ge.patch
- v2-0008-VERY-WIP-parallel-doc-generation.patch
On 20.03.23 03:33, Andres Freund wrote: >> I did end up getting stuck when hacking on this, and ended up adding css >> support for nochunk and support for the website style for htmlhelp and >> nochunk, as well as obsoleting the need for copying the css files... But >> perhaps that's a bit too much. > Updated set of patches attached. This one works in older meson versions too > and adds install-world and install-quiet targets. Oh, this patch set grew quite quickly. ;-) [PATCH v2 1/8] meson: rename html_help target to htmlhelp This is obvious. [PATCH v2 5/8] docs: html: copy images to output as part of xslt build Making the XSLT stylesheets do the copying has some appeal. I think it would only work for SVG (or other XML) files, which I guess is okay, but maybe the templates should have a filter on format="SVG" or something. Also, this copying actually modifies the files in some XML-equivalent way. Also okay, I think, but worth noting. Note sure why you removed this comment -<!-- strip directory name from image filerefs --> since the code still exists. [PATCH v2 6/8] wip: docs: copy or inline css This seems pretty complicated compared to just copying a file?
Hi, On 2023-03-20 11:58:08 +0100, Peter Eisentraut wrote: > Oh, this patch set grew quite quickly. ;-) Yep :) > [PATCH v2 5/8] docs: html: copy images to output as part of xslt build > > Making the XSLT stylesheets do the copying has some appeal. I think it > would only work for SVG (or other XML) files, which I guess is okay, but > maybe the templates should have a filter on format="SVG" or something. Also, > this copying actually modifies the files in some XML-equivalent way. Also > okay, I think, but worth noting. I think it can be made work for non-xml files with xinclude too. But the restriction around only working in top-level stylesheets (vs everywhere for documents) is quite annoying. > [PATCH v2 6/8] wip: docs: copy or inline css > > This seems pretty complicated compared to just copying a file? Mainly that it works correctly for the standalone file. Greetings, Andres Freund
Hi, On 2023-03-19 19:33:38 -0700, Andres Freund wrote: > I think we can make the docs build in parallel and incrementally, by building > the different parts of the docs in parallel, using --stringparam rootid, > e.g. building each 'part' separately. > > A very very rough draft attached: > > parallel with parts: > real 0m10.831s > user 0m58.295s > sys 0m1.402s > > normal: > real 0m32.215s > user 0m31.876s > sys 0m0.328s > > 1/3 of the build time at 2x the cost is nothing to sneeze at. I could not make myself stop trying to figure out where the big constant time factor comes from. Every invocation costs about 2s, even if not much is rendered. Turns out, that's solely spent building all the <xsl:key>s. The first time *any* key() is invoked for a document, all the keys are computed in a single pass over the document. A single reasonable key doesn't take that much time, even for the size of our docs. But there are several redundant keys being built. Some of them somewhat expensive. E.g. each <xsl:key name="genid" match="*" use="generate-id()"/> takes about 300ms. There's one in chunk-common and one in docbook-no-doctype.xsl. I'm going to cry now. Greetings, Andres Freund
Hi, On 2023-03-20 10:32:49 -0700, Andres Freund wrote: > On 2023-03-20 11:58:08 +0100, Peter Eisentraut wrote: > > Oh, this patch set grew quite quickly. ;-) > > Yep :) Unless somebody sees a reason to wait, I am planning to commit: meson: add install-{quiet, world} targets meson: add install-{docs,doc-html,doc-man} targets meson: make install_test_files more generic, rename to install_files While I don't think we have necessarily the path forward around .css and .svg, the above are independent of that. For the .svg: I wonder if we should just inline the images in the chunked html, just like we do in the single page one. It's not like we reuse one image across a lot of pages, so there's no bandwidth saved from having the images separate... For the .css: docbook-xsl actually has support for writing the .css: [1] - but it requires the .css file be valid xml. I wonder if the cleanest approch would be to have a build step to create .css.xml - then the non-chunked build's generate.css.header would do the right thing. I'll start a new thread for docs: speed up docs build by special-casing the gentext.template VERY WIP: parallel doc generation after the feature freeze. After looking into it a tiny bit more, it seems we should use neither pandoc nor dbtoepub for epub generation. All the dbtoepub does is to invoke the docbook-xsl support for epubs and zip the result - except it doesn't use our stylesheets, so it looks randomly different and doesn't use our speedups. At the very least we should use our customizations, if we want epub support. Or we should just remove it. Pandoc unfortunately doesn't do docbook well enough to be usable for now to directly parse our docbook. Regards, Andres [1] https://docbook.sourceforge.net/release/xsl/current/doc/html/custom.css.source.html
Hi, On 2023-03-22 11:59:17 -0700, Andres Freund wrote: > Unless somebody sees a reason to wait, I am planning to commit: > meson: add install-{quiet, world} targets > meson: add install-{docs,doc-html,doc-man} targets > meson: make install_test_files more generic, rename to install_files I've done that now. > For the .css: docbook-xsl actually has support for writing the .css: [1] - but > it requires the .css file be valid xml. I wonder if the cleanest approch would > be to have a build step to create .css.xml - then the non-chunked build's > generate.css.header would do the right thing. We don't even need to do that! The attached patch just creates a wrapper css.xml that loads the .css via an entity reference. I think this looks reasonably complicated, given that it gives us a working stylesheet for the non-chunked output? I don't know if my hack of putting the paramters in stylesheet-common.xsl is reasonable. Perhaps we should just include stylesheet-html-common.xsl in stylesheet-hh.xsl, then this uglyness wouldn't be required. Greetings, Andres Freund
Attachment
On 24.03.23 08:26, Andres Freund wrote: >> For the .css: docbook-xsl actually has support for writing the .css: [1] - but >> it requires the .css file be valid xml. I wonder if the cleanest approch would >> be to have a build step to create .css.xml - then the non-chunked build's >> generate.css.header would do the right thing. > > We don't even need to do that! The attached patch just creates a wrapper > css.xml that loads the .css via an entity reference. That looks like a better solution. > I don't know if my hack of putting the paramters in stylesheet-common.xsl is > reasonable. Perhaps we should just include stylesheet-html-common.xsl in > stylesheet-hh.xsl, then this uglyness wouldn't be required. Maybe, but it's not clear whether all the customizations in there are applicable to htmlhelp. Another option here is to remove support for htmlhelp.
Remove 'htmlhelp' documentat format (was meson documentation build open issues)
From
Andres Freund
Date:
Hi, On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote: > Another option here is to remove support for htmlhelp. That might actually be the best path - it certainly doesn't look like anybody has been actively using it. Or otherwise somebody would have complained about there not being any instructions on how to actually compile a .chm file. And perhaps complained that it takes next to forever to build. I also have the impression that people don't use the .chm stuff much anymore, but that might just be me not using windows. Greetings, Andres
Re: Remove 'htmlhelp' documentat format (was meson documentation build open issues)
From
Daniel Gustafsson
Date:
> On 24 Mar 2023, at 17:58, Andres Freund <andres@anarazel.de> wrote: > On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote: >> Another option here is to remove support for htmlhelp. > > That might actually be the best path - it certainly doesn't look like anybody > has been actively using it. I had no idea we had support for building a .chm until reading this, but I've also never come across anyone asking for such a docset. FWIW, no objections to it going. -- Daniel Gustafsson
Re: Remove 'htmlhelp' documentat format (was meson documentation build open issues)
From
Peter Eisentraut
Date:
On 24.03.23 17:58, Andres Freund wrote: > On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote: >> Another option here is to remove support for htmlhelp. > > That might actually be the best path - it certainly doesn't look like anybody > has been actively using it. Or otherwise somebody would have complained about > there not being any instructions on how to actually compile a .chm file. And > perhaps complained that it takes next to forever to build. > > I also have the impression that people don't use the .chm stuff much anymore, > but that might just be me not using windows. I think in ancient times, pgadmin used it for its internal help. But I have heard less about htmlhelp over the years than about the info format.
Re: Remove 'htmlhelp' documentat format (was meson documentation build open issues)
From
Dave Page
Date:
On Tue, 28 Mar 2023 at 10:46, Peter Eisentraut <peter.eisentraut@enterprisedb.com> wrote:
On 24.03.23 17:58, Andres Freund wrote:
> On 2023-03-24 11:59:23 +0100, Peter Eisentraut wrote:
>> Another option here is to remove support for htmlhelp.
>
> That might actually be the best path - it certainly doesn't look like anybody
> has been actively using it. Or otherwise somebody would have complained about
> there not being any instructions on how to actually compile a .chm file. And
> perhaps complained that it takes next to forever to build.
>
> I also have the impression that people don't use the .chm stuff much anymore,
> but that might just be me not using windows.
I think in ancient times, pgadmin used it for its internal help.
Yes, very ancient :-). We use Sphinx now.
But I have heard less about htmlhelp over the years than about the info
format.
On 15.03.23 08:14, Peter Eisentraut wrote: > I have identified several open issues with the documentation build under > Meson (approximately in priority order): Some work has been done on this. Here is my current assessment. > 1. Image files are not handled at all, so they don't show up in the > final product. This is fixed. > 2. Defaults to website stylesheet, no way to configure. This should be > adjusted to match the make build. This is fixed. > 3. The various build targets and their combinations are mismatching and > incomplete. This has been improved, and I see there is documentation. I think it's still an issue that "make docs" builds html and man but "ninja docs" only builds html. For some reason the wiki page actually claims that ninja docs builds both, but this does not happen for me. > 4. There doesn't appear to be a way to install the documentation. This has been addressed. > 5. There doesn't appear to be an equivalent of "make world" and "make > install-world" that includes documentation builds. This has been addressed with the additional meson auto options. But it seems that these options only control building, not installing, so there is no "install-world" aspect yet.
Hi, On 2023-04-05 12:24:04 +0200, Peter Eisentraut wrote: > On 15.03.23 08:14, Peter Eisentraut wrote: > > 3. The various build targets and their combinations are mismatching and > > incomplete. > > This has been improved, and I see there is documentation. > > I think it's still an issue that "make docs" builds html and man but "ninja > docs" only builds html. For some reason the wiki page actually claims that > ninja docs builds both, but this does not happen for me. It used to, but Tom insisted that it should not. I'm afraid that it's not quite possible to emulate make here. 'make docs' at the toplevel builds both HTML and manpages. But 'make -C doc/src/sgml', only builds HTML. > > 5. There doesn't appear to be an equivalent of "make world" and "make > > install-world" that includes documentation builds. > > This has been addressed with the additional meson auto options. But it > seems that these options only control building, not installing, so there is > no "install-world" aspect yet. I'm not following - install-world install docs if the docs feature is available, and not if not? Greetings, Andres Freund
On 05.04.23 16:45, Andres Freund wrote: >> I think it's still an issue that "make docs" builds html and man but "ninja >> docs" only builds html. For some reason the wiki page actually claims that >> ninja docs builds both, but this does not happen for me. > > It used to, but Tom insisted that it should not. I'm afraid that it's not > quite possible to emulate make here. 'make docs' at the toplevel builds both > HTML and manpages. But 'make -C doc/src/sgml', only builds HTML. Ok, not a topic for this thread then. >>> 5. There doesn't appear to be an equivalent of "make world" and "make >>> install-world" that includes documentation builds. >> >> This has been addressed with the additional meson auto options. But it >> seems that these options only control building, not installing, so there is >> no "install-world" aspect yet. > > I'm not following - install-world install docs if the docs feature is > available, and not if not? I had expected that if meson setup enables the 'docs' feature, then meson compile will build the documentation, which happens, and meson install will install it, which does not happen.
On 2023-04-06 Th 05:11, Peter Eisentraut wrote:
On 05.04.23 16:45, Andres Freund wrote:I think it's still an issue that "make docs" builds html and man but "ninja
docs" only builds html. For some reason the wiki page actually claims that
ninja docs builds both, but this does not happen for me.
It used to, but Tom insisted that it should not. I'm afraid that it's not
quite possible to emulate make here. 'make docs' at the toplevel builds both
HTML and manpages. But 'make -C doc/src/sgml', only builds HTML.
Ok, not a topic for this thread then.5. There doesn't appear to be an equivalent of "make world" and "make
install-world" that includes documentation builds.
This has been addressed with the additional meson auto options. But it
seems that these options only control building, not installing, so there is
no "install-world" aspect yet.
I'm not following - install-world install docs if the docs feature is
available, and not if not?
I had expected that if meson setup enables the 'docs' feature, then meson compile will build the documentation, which happens, and meson install will install it, which does not happen.
"meson compile" doesn't seem to build the docs by default ( see <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2023-04-06%2018%3A17%3A04&stg=build>), and I'd rather it didn't, building the docs is a separate and optional step for the buildfarm.
cheers
andrew
-- Andrew Dunstan EDB: https://www.enterprisedb.com
On 07.04.23 16:39, Andrew Dunstan wrote: >>>>> 5. There doesn't appear to be an equivalent of "make world" and "make >>>>> install-world" that includes documentation builds. >>>> >>>> This has been addressed with the additional meson auto options. But it >>>> seems that these options only control building, not installing, so >>>> there is >>>> no "install-world" aspect yet. >>> >>> I'm not following - install-world install docs if the docs feature is >>> available, and not if not? >> >> I had expected that if meson setup enables the 'docs' feature, then >> meson compile will build the documentation, which happens, and meson >> install will install it, which does not happen. > > "meson compile" doesn't seem to build the docs by default ( see > <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2023-04-06%2018%3A17%3A04&stg=build>), and I'drather it didn't, building the docs is a separate and optional step for the buildfarm. You can control this with the "docs" option for meson, as of recently.
Re: Peter Eisentraut > > "meson compile" doesn't seem to build the docs by default ( see <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2023-04-06%2018%3A17%3A04&stg=build>), > > and I'd rather it didn't, building the docs is a separate and optional > > step for the buildfarm. > > You can control this with the "docs" option for meson, as of recently. I've been looking into switching the Debian PG 17 build to meson, but I'm running into several problems. * The docs are still not built by default, and -Ddocs=enabled doesn't change that * None of the "build docs" targets are documented in install-meson.html * "ninja -C build alldocs" works, but it's impossible to see what flavors it's actually building. Everything is autodetected, and perhaps I would like to no build the .txt/something variants, but I have no idea what switch that is, or what package I have to uninstall so it's not autodetected (only html and pdf are documented.) Are there any other targets for the individual formats? (I could probably use one for the manpages only, without the html.) Non-doc issues: * LLVM is off by default (ok), when I enable it with -Dllvm=auto, it gets detected, but no .bc files are built, nor installed * selinux is not autodetected. It needs -Dselinux=auto, but that's not documented in install-meson.html * There is no split between libdir and pkglibdir. We had used that in the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff -> /usr/lib/postgresql/17/lib. Christoph
Hi, On 2023-11-03 15:26:05 +0100, Christoph Berg wrote: > Re: Peter Eisentraut > > > "meson compile" doesn't seem to build the docs by default ( see <https://buildfarm.postgresql.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2023-04-06%2018%3A17%3A04&stg=build>), > > > and I'd rather it didn't, building the docs is a separate and optional > > > step for the buildfarm. > > > > You can control this with the "docs" option for meson, as of recently. > > I've been looking into switching the Debian PG 17 build to meson, but > I'm running into several problems. > > * The docs are still not built by default, and -Ddocs=enabled doesn't > change that Maybe I am missing something - they aren't built by default in autoconf either? > * None of the "build docs" targets are documented in install-meson.html Hm, odd, I thought they were, but you are right. There were some docs patches that we never really could find agreement upon :/ > * "ninja -C build alldocs" works, but it's impossible to see what > flavors it's actually building. Everything is autodetected, and > perhaps I would like to no build the .txt/something variants, > but I have no idea what switch that is, or what package I have to > uninstall so it's not autodetected (only html and pdf are > documented.) I think a package build should probably turn off auto-detection ( meson setup --auto-features=disabled) and enable specific features that are desired - in which case you get errors if they are not available. Which presumably is the behaviour you'd like? > Are there any other targets for the individual formats? (I could > probably use one for the manpages only, without the html.) Yes, there are. ninja doc/src/sgml/{postgres-A4.pdf,html,postgres.html,man1} Perhaps more interesting for your purposes, there are the install-doc-{html,man} targets. I remember discussing adding doc-{html,man} targets alongside install-doc-{html,man}, not sure why we ended up not doing that. I'd be in favor of adding them. I've also been wondering about a 'help' target that documents important targets in a interactively usable way. > Non-doc issues: > > * LLVM is off by default (ok), when I enable it with -Dllvm=auto, it > gets detected, but no .bc files are built, nor installed Support for that has not yet been merged. > * selinux is not autodetected. It needs -Dselinux=auto, but that's not > documented in install-meson.html Uh, huh. There's no documentation for --with-selinux in the installation.sgml either, just in sepgsql.sgml. So when the relevant docs got translated to meson, -Dselinux= wasn't documented either. > * There is no split between libdir and pkglibdir. We had used that in > the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff -> > /usr/lib/postgresql/17/lib. I don't think the autoconf build currently exposes separately configuring pkglibdir either, I think that's a debian patch? I'm entirely open to adding an explicit configuration option for this though. Thanks for looking at this, it's quite helpful! Greetings, Andres Freund
Re: Andres Freund > > > You can control this with the "docs" option for meson, as of recently. > > > > I've been looking into switching the Debian PG 17 build to meson, but > > I'm running into several problems. > > > > * The docs are still not built by default, and -Ddocs=enabled doesn't > > change that > > Maybe I am missing something - they aren't built by default in autoconf > either? True, but the documentation (and this thread) reads like it should. Or at least it should, when I explicitly say -Ddocs=enabled. What would also help is when the tail of the meson output had a list of features that are enabled. There's the list of "External libraries" which is quite helpful at figuring out what's still missing, but perhaps this could be extended: Features LLVM : YES (/usr/bin/llvm-config-16) DOCS : YES (html pdf texinfo) Atm it's hidden in the long initial blurb of "Checking for.." and the "NO" in there don't really stand out as much, since some of them are normal. > > * "ninja -C build alldocs" works, but it's impossible to see what > > flavors it's actually building. Everything is autodetected, and > > perhaps I would like to no build the .txt/something variants, > > but I have no idea what switch that is, or what package I have to > > uninstall so it's not autodetected (only html and pdf are > > documented.) > > I think a package build should probably turn off auto-detection ( > meson setup --auto-features=disabled) and enable specific features that are > desired - in which case you get errors if they are not available. Which > presumably is the behaviour you'd like? I'm still trying to figure out the best spot in that space of options. Currently I'm still in the phase of getting it to work at all; the end result might well use that option. > > Are there any other targets for the individual formats? (I could > > probably use one for the manpages only, without the html.) > > Yes, there are. > ninja doc/src/sgml/{postgres-A4.pdf,html,postgres.html,man1} Oh, that was not obvious to me that this "make $some_file" style command would work. (But it still leaves the problem of knowing which targets there are.) > Perhaps more interesting for your purposes, there are the > install-doc-{html,man} targets. Hmm, I thought I had tried these, but apparently managed to miss them. Thanks. install-doc-man seems to install "man1" only, though? (It seems to compile man5/man7, but not install them.) > I remember discussing adding doc-{html,man} targets alongside > install-doc-{html,man}, not sure why we ended up not doing that. I'd be in > favor of adding them. > > I've also been wondering about a 'help' target that documents important > targets in a interactively usable way. That is definitely missing, yes. I found out about "alldocs" only after reading the meson files, and that took more than it should have. > > Non-doc issues: > > > > * LLVM is off by default (ok), when I enable it with -Dllvm=auto, it > > gets detected, but no .bc files are built, nor installed > > Support for that has not yet been merged. Oh, that's a showstopper. I thought meson would already be ready for production use. There is indeed an "experimental" note in install-requirements.html, but not in install-meson.html > > * selinux is not autodetected. It needs -Dselinux=auto, but that's not > > documented in install-meson.html > > Uh, huh. There's no documentation for --with-selinux in the installation.sgml > either, just in sepgsql.sgml. So when the relevant docs got translated to > meson, -Dselinux= wasn't documented either. Ok. It does show up in "External libraries" and was enabled in the Debian packages before. Why isn't it "auto" like the others? > > * There is no split between libdir and pkglibdir. We had used that in > > the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff -> > > /usr/lib/postgresql/17/lib. > > I don't think the autoconf build currently exposes separately configuring > pkglibdir either, I think that's a debian patch? I'm entirely open to adding > an explicit configuration option for this though. That would definitely be helpful. > Thanks for looking at this, it's quite helpful! Thanks for the feedback! Christoph
Hi, On 2023-11-03 19:19:17 +0100, Christoph Berg wrote: > Re: Andres Freund > > > > You can control this with the "docs" option for meson, as of recently. > > > > > > I've been looking into switching the Debian PG 17 build to meson, but > > > I'm running into several problems. > > > > > > * The docs are still not built by default, and -Ddocs=enabled doesn't > > > change that > > > > Maybe I am missing something - they aren't built by default in autoconf > > either? > > True, but the documentation (and this thread) reads like it should. Or > at least it should, when I explicitly say -Ddocs=enabled. My understanding of the intent of the options is to make meson error out if the required dependencies are not available, not that it controls when the build targets are built. The reason for that is simply that the docs take too long to build. > What would also help is when the tail of the meson output had a list > of features that are enabled. There's the list of "External libraries" > which is quite helpful at figuring out what's still missing, but > perhaps this could be extended: > > Features > LLVM : YES (/usr/bin/llvm-config-16) > DOCS : YES (html pdf texinfo) > > Atm it's hidden in the long initial blurb of "Checking for.." and the > "NO" in there don't really stand out as much, since some of them are > normal. The summary does include both. LLVM is 'llvm', man/html docs is 'docs' and pdf docs as 'docs_pdf'. > > > Are there any other targets for the individual formats? (I could > > > probably use one for the manpages only, without the html.) > > > > Yes, there are. > > ninja doc/src/sgml/{postgres-A4.pdf,html,postgres.html,man1} > > Oh, that was not obvious to me that this "make $some_file" style > command would work. (But it still leaves the problem of knowing which > targets there are.) Yes, you can trigger building any file that way. The following is *not* an argument the docs targets shouldn't be documented (working on a patch), just something that might be helpful until then / separately. You can see which targets are built with ninja -t targets all|grep doc/src/ > > Perhaps more interesting for your purposes, there are the > > install-doc-{html,man} targets. > > Hmm, I thought I had tried these, but apparently managed to miss them. > Thanks. > > install-doc-man seems to install "man1" only, though? > (It seems to compile man5/man7, but not install them.) Ugh, that's obviously a bug. I'll fix it. > > > Non-doc issues: > > > > > > * LLVM is off by default (ok), when I enable it with -Dllvm=auto, it > > > gets detected, but no .bc files are built, nor installed > > > > Support for that has not yet been merged. > > Oh, that's a showstopper. I thought meson would already be ready for > production use. There is indeed an "experimental" note in > install-requirements.html, but not in install-meson.html I'm working on merging it. Having it for core PG isn't a huge difficulty, the extension story is what's been holding me back... > > > * selinux is not autodetected. It needs -Dselinux=auto, but that's not > > > documented in install-meson.html > > > > Uh, huh. There's no documentation for --with-selinux in the installation.sgml > > either, just in sepgsql.sgml. So when the relevant docs got translated to > > meson, -Dselinux= wasn't documented either. > > Ok. It does show up in "External libraries" and was enabled in the > Debian packages before. > > Why isn't it "auto" like the others? I don't really remember why I did that, but it's platform specific, maybe that's why I did it that way? > > > * There is no split between libdir and pkglibdir. We had used that in > > > the past for libpq -> /usr/lib/x86_64-linux-gnu and PG stuff -> > > > /usr/lib/postgresql/17/lib. > > > > I don't think the autoconf build currently exposes separately configuring > > pkglibdir either, I think that's a debian patch? I'm entirely open to adding > > an explicit configuration option for this though. > > That would definitely be helpful. I have a patch locally, will send it together with a few others in a bit. Greetings, Andres Freund
Re: Andres Freund > The reason for that is simply that the docs take too long to build. That why I'd prefer to be able to separate arch:all and arch:any builds, yes. > The summary does include both. LLVM is 'llvm', man/html docs is 'docs' and pdf > docs as 'docs_pdf'. Sorry, I should have looked closer. :( > The following is *not* an argument the docs targets shouldn't be documented > (working on a patch), just something that might be helpful until then / > separately. You can see which targets are built with > > ninja -t targets all|grep doc/src/ Thanks. > > Oh, that's a showstopper. I thought meson would already be ready for > > production use. There is indeed an "experimental" note in > > install-requirements.html, but not in install-meson.html > > I'm working on merging it. Having it for core PG isn't a huge difficulty, the > extension story is what's been holding me back... In-core extensions or external ones? > > Why isn't it "auto" like the others? > > I don't really remember why I did that, but it's platform specific, maybe > that's why I did it that way? Isn't that kind the point of autodetecting things? Aren't bonjour and bsd_auth autodetected as well? > > > I don't think the autoconf build currently exposes separately configuring > > > pkglibdir either, I think that's a debian patch? I'm entirely open to adding > > > an explicit configuration option for this though. > > > > That would definitely be helpful. > > I have a patch locally, will send it together with a few others in a bit. Thanks! Christoph
Hi, On 2023-11-03 20:19:18 +0100, Christoph Berg wrote: > Re: Andres Freund > > The reason for that is simply that the docs take too long to build. > > That why I'd prefer to be able to separate arch:all and arch:any > builds, yes. What's stopping you from doing that? I think the only arch:any content we have is the docs, and those you can build separately? Doc builds do trigger generation of a handful of files besides the docs, but not more. > > > Oh, that's a showstopper. I thought meson would already be ready for > > > production use. There is indeed an "experimental" note in > > > install-requirements.html, but not in install-meson.html > > > > I'm working on merging it. Having it for core PG isn't a huge difficulty, the > > extension story is what's been holding me back... > > In-core extensions or external ones? Both, although the difficulty of doing it is somewhat separate for each. > > > Why isn't it "auto" like the others? > > > > I don't really remember why I did that, but it's platform specific, maybe > > that's why I did it that way? > > Isn't that kind the point of autodetecting things? Aren't bonjour and > bsd_auth autodetected as well? I'd be happy to change it, unless somebody objects? > > > > I don't think the autoconf build currently exposes separately configuring > > > > pkglibdir either, I think that's a debian patch? I'm entirely open to adding > > > > an explicit configuration option for this though. > > > > > > That would definitely be helpful. > > > > I have a patch locally, will send it together with a few others in a bit. > > Thanks! Attached. 0001 - the bugfix for install-man only installing man1, I'll push that soon 0002 - Document --with-selinux/-Dselinux options centrally 0003 - Add doc-{html,man} targets I'm not quite sure it's worth it, but it's basically free, so ... 0004 - Documentation for important build targets I'm not entirely happy with the formatting, but it looks like that's mostly a CSS issue. I started a thread on fixing that on -www. 0005 - Add -Dpkglibdir option I guess we might want to do the same for configure if we decide to do this? Greetings, Andres Freund
Attachment
Re: Andres Freund > > > The reason for that is simply that the docs take too long to build. > > > > That why I'd prefer to be able to separate arch:all and arch:any > > builds, yes. > > What's stopping you from doing that? I think the only arch:any content we > have is the docs, and those you can build separately? Doc builds do trigger > generation of a handful of files besides the docs, but not more. Historically, .deb files have been required to contain the manpages for all executables even when there's a separate -doc package. This means we'd need a separate (hopefully fast) manpage build even when the arch:any binaries are built. We might get around that by introducing a new postgresql-manpages-XX arch:all package, but that might be too much micropackaging. The install-doc-man target will likely fix it, will play with it a bit more, thanks. > > > I'm working on merging it. Having it for core PG isn't a huge difficulty, the > > > extension story is what's been holding me back... > > > > In-core extensions or external ones? > > Both, although the difficulty of doing it is somewhat separate for each. I'd think most external extensions could stay with pgxs.mk for the time being. > + <varlistentry id="configure-with-sepgsql-meson"> > + <term><option>-Dselinux={ disabled | auto | enabled }</option></term> > + <listitem> > + <para> > + Build with selinux support, enabling the <xref linkend="sepgsql"/> > + extension. This option defaults to ... auto? > index 90e2c062fa8..003b57498bb 100644 > --- a/doc/src/sgml/meson.build > +++ b/doc/src/sgml/meson.build > @@ -142,6 +142,7 @@ if docs_dep.found() > '--install-dir-contents', dir_doc_html, html], > build_always_stale: true, build_by_default: false, > ) > + alias_target('doc-html', install_doc_html) > alias_target('install-doc-html', install_doc_html) Shouldn't this just build the html docs, without installing? > + alias_target('doc-man', install_doc_html) > alias_target('install-doc-man', install_doc_man) ... same > + <varlistentry id="meson-target-install-world"> > + <term><option>install-install-world</option></term> install-world > + <varlistentry id="meson-target-install-doc-man"> > + <term><option>install-doc-html</option></term> > + <listitem> > + <para> > + Install documentation in man page format. install-doc-man > + <sect3 id="meson-targets-docs"> > + <title>Documentation Targets</title> > + <varlistentry id="meson-target-docs"> > + <term><option>docs</option></term> > + <term><option>doc-html</option></term> > + <listitem> > + <para> > + Build documentation in multi-page HTML format. Note that > + <option>docs</option> does <emphasis>not</emphasis> include building > + man page documentation, as man page generation seldom fails when > + building HTML documentation succeeds. Why is that a reason for not building the manpages? > + <sect3 id="meson-targets-code"> > + <title>Code Targets</title> I would have expected the sections to be in the order build-docs-install. Having install first seems weird to me. > + <sect3 id="meson-targets-other"> > + <title>Other Targets</title> > + > + <variablelist> > + > + <varlistentry id="meson-target-clean"> > + <term><option>clean</option></term> > + <listitem> > + <para> > + Remove all build products > + </para> > + </listitem> > + </varlistentry> > + > + <varlistentry id="meson-target-test"> > + <term><option>test</option></term> > + <listitem> > + <para> > + Remove all enabled tests. Support for some classes of tests can be > + enabled / disabled with <xref linkend="configure-tap-tests-meson"/> > + and <xref linkend="configure-pg-test-extra-meson"/>. This should explicitly say if contrib tests are included (or there needs to be a separate test-world target.) > Subject: [PATCH v1 5/5] meson: Add -Dpkglibdir option Will give that a try, thanks! Christoph
On 03.11.23 22:16, Andres Freund wrote: [selinux option] >>>> Why isn't it "auto" like the others? >>> I don't really remember why I did that, but it's platform specific, maybe >>> that's why I did it that way? >> Isn't that kind the point of autodetecting things? Aren't bonjour and >> bsd_auth autodetected as well? > I'd be happy to change it, unless somebody objects? Makes sense to me to change it to auto.
On 03.11.23 19:19, Christoph Berg wrote: >>>> You can control this with the "docs" option for meson, as of recently. >>> I've been looking into switching the Debian PG 17 build to meson, but >>> I'm running into several problems. >>> >>> * The docs are still not built by default, and -Ddocs=enabled doesn't >>> change that >> Maybe I am missing something - they aren't built by default in autoconf >> either? > True, but the documentation (and this thread) reads like it should. Or > at least it should, when I explicitly say -Ddocs=enabled. > > What would also help is when the tail of the meson output had a list > of features that are enabled. There's the list of "External libraries" > which is quite helpful at figuring out what's still missing, but > perhaps this could be extended: > > Features > LLVM : YES (/usr/bin/llvm-config-16) > DOCS : YES (html pdf texinfo) > > Atm it's hidden in the long initial blurb of "Checking for.." and the > "NO" in there don't really stand out as much, since some of them are > normal. I don't feel like we have fully worked out how the docs options should fit together. With the make build system, there is a canonical sequence of make world make check-world make install-world that encompasses everything. Now with meson to handle the documentation one needs to remember a variety of additional targets. (There is a risk that once this gets more widespread, more people will submit broken documentation.) I would like to have some set of options that enables it so that the standard documentation targets become part of "meson compile" and "meson install".
Hi, On November 7, 2023 7:55:37 AM PST, Peter Eisentraut <peter@eisentraut.org> wrote: >On 03.11.23 19:19, Christoph Berg wrote: >>>>> You can control this with the "docs" option for meson, as of recently. >>>> I've been looking into switching the Debian PG 17 build to meson, but >>>> I'm running into several problems. >>>> >>>> * The docs are still not built by default, and -Ddocs=enabled doesn't >>>> change that >>> Maybe I am missing something - they aren't built by default in autoconf >>> either? >> True, but the documentation (and this thread) reads like it should. Or >> at least it should, when I explicitly say -Ddocs=enabled. >> >> What would also help is when the tail of the meson output had a list >> of features that are enabled. There's the list of "External libraries" >> which is quite helpful at figuring out what's still missing, but >> perhaps this could be extended: >> >> Features >> LLVM : YES (/usr/bin/llvm-config-16) >> DOCS : YES (html pdf texinfo) >> >> Atm it's hidden in the long initial blurb of "Checking for.." and the >> "NO" in there don't really stand out as much, since some of them are >> normal. > >I don't feel like we have fully worked out how the docs options should fit together. > >With the make build system, there is a canonical sequence of > >make world >make check-world >make install-world > >that encompasses everything. > >Now with meson to handle the documentation one needs to remember a variety of additional targets. (There is a risk thatonce this gets more widespread, more people will submit broken documentation.) install-world with meson also installs docs. >I would like to have some set of options that enables it so that the standard documentation targets become part of "mesoncompile" and "meson install". -0.5 - it's just too painfully slow. For all scripted uses you can just as well use install-world... Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Andres Freund <andres@anarazel.de> writes: > On November 7, 2023 7:55:37 AM PST, Peter Eisentraut <peter@eisentraut.org> wrote: >> I would like to have some set of options that enables it so that the standard documentation targets become part of "mesoncompile" and "meson install". > -0.5 - it's just too painfully slow. For all scripted uses you can just as well use install-world... I think we should set up the meson stuff so that "install" and "install-world" cover exactly what they did with "make". Otherwise there will be too much confusion. regards, tom lane
On 2023-Nov-07, Andres Freund wrote: > >I would like to have some set of options that enables it so that the > >standard documentation targets become part of "meson compile" and > >"meson install". > > -0.5 - it's just too painfully slow. For all scripted uses you can just as well use install-world... If the problem is broken doc patches, then maybe a solution is to include the `xmllint --noout --valid` target in whatever the check-world equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't seem to do that anywhere. Doing the no-output lint run is very fast (375ms real time in my machine, whereas "make html" takes 27s). -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "¿Cómo puedes confiar en algo que pagas y que no ves, y no confiar en algo que te dan y te lo muestran?" (Germán Poo)
Alvaro Herrera <alvherre@alvh.no-ip.org> writes: > If the problem is broken doc patches, then maybe a solution is to > include the `xmllint --noout --valid` target in whatever the check-world > equivalent is for meson. +1, but let's do that for the "make" build too. I see that doc/src/sgml/Makefile has a "check" target, but AFAICS it's not wired up to the top-level check-world. regards, tom lane
Hi, On 2023-11-06 10:45:27 +0100, Christoph Berg wrote: > Re: Andres Freund > > > > The reason for that is simply that the docs take too long to build. > > > > > > That why I'd prefer to be able to separate arch:all and arch:any > > > builds, yes. > > > > What's stopping you from doing that? I think the only arch:any content we > > have is the docs, and those you can build separately? Doc builds do trigger > > generation of a handful of files besides the docs, but not more. > > Historically, .deb files have been required to contain the manpages > for all executables even when there's a separate -doc package. This > means we'd need a separate (hopefully fast) manpage build even when > the arch:any binaries are built. Manpages are a bit faster to build than html, but not a whole lot. Both are a lot faster than PDF. > We might get around that by introducing a new postgresql-manpages-XX > arch:all package, but that might be too much micropackaging. I've not done packaging in, uh, a fair while, but isn't the common solution to that a -common package? There might be a few more files we could put itno one. > > + <varlistentry id="configure-with-sepgsql-meson"> > > + <term><option>-Dselinux={ disabled | auto | enabled }</option></term> > > + <listitem> > > + <para> > > + Build with selinux support, enabling the <xref linkend="sepgsql"/> > > + extension. > > This option defaults to ... auto? Not quite sure what you mean? Today it defaults to disabled, a patch changing that should also change the docs? > > index 90e2c062fa8..003b57498bb 100644 > > --- a/doc/src/sgml/meson.build > > +++ b/doc/src/sgml/meson.build > > @@ -142,6 +142,7 @@ if docs_dep.found() > > '--install-dir-contents', dir_doc_html, html], > > build_always_stale: true, build_by_default: false, > > ) > > + alias_target('doc-html', install_doc_html) > > alias_target('install-doc-html', install_doc_html) > > Shouldn't this just build the html docs, without installing? > > > + alias_target('doc-man', install_doc_html) > > alias_target('install-doc-man', install_doc_man) > > ... same > > > > + <varlistentry id="meson-target-install-world"> > > + <term><option>install-install-world</option></term> > > install-world > > > + <varlistentry id="meson-target-install-doc-man"> > > + <term><option>install-doc-html</option></term> > > + <listitem> > > + <para> > > + Install documentation in man page format. > > install-doc-man Oops. > > + <sect3 id="meson-targets-docs"> > > + <title>Documentation Targets</title> > > > + <varlistentry id="meson-target-docs"> > > + <term><option>docs</option></term> > > + <term><option>doc-html</option></term> > > + <listitem> > > + <para> > > + Build documentation in multi-page HTML format. Note that > > + <option>docs</option> does <emphasis>not</emphasis> include building > > + man page documentation, as man page generation seldom fails when > > + building HTML documentation succeeds. > > Why is that a reason for not building the manpages? I didn't have it that way, and Tom argued strongly for maintaining that behaviour from the make build. Personally I wouldn't. > > + <sect3 id="meson-targets-code"> > > + <title>Code Targets</title> > > I would have expected the sections to be in the order > build-docs-install. Having install first seems weird to me. Makes sense to me. I just had the install first because I wrote it first because of our conversation... > > + <sect3 id="meson-targets-other"> > > + <title>Other Targets</title> > > + > > + <variablelist> > > + > > + <varlistentry id="meson-target-clean"> > > + <term><option>clean</option></term> > > + <listitem> > > + <para> > > + Remove all build products > > + </para> > > + </listitem> > > + </varlistentry> > > + > > + <varlistentry id="meson-target-test"> > > + <term><option>test</option></term> > > + <listitem> > > + <para> > > + Remove all enabled tests. Support for some classes of tests can be > > + enabled / disabled with <xref linkend="configure-tap-tests-meson"/> > > + and <xref linkend="configure-pg-test-extra-meson"/>. > > This should explicitly say if contrib tests are included (or there > needs to be a separate test-world target.) They are included, will state that. And also s/Remove/Run/ > > Subject: [PATCH v1 5/5] meson: Add -Dpkglibdir option > > Will give that a try, thanks! Thanks for the review! Greetings, Andres Freund
Re: Andres Freund > > We might get around that by introducing a new postgresql-manpages-XX > > arch:all package, but that might be too much micropackaging. > > I've not done packaging in, uh, a fair while, but isn't the common solution to > that a -common package? There might be a few more files we could put itno one. True. /usr/share/postgresql/17/ is 4.2MB here, with 1.5MB manpages, 1.1MB /extensions/ and some other bits. Will consider, thanks. > > > + <varlistentry id="configure-with-sepgsql-meson"> > > > + <term><option>-Dselinux={ disabled | auto | enabled }</option></term> > > > + <listitem> > > > + <para> > > > + Build with selinux support, enabling the <xref linkend="sepgsql"/> > > > + extension. > > > > This option defaults to ... auto? > > Not quite sure what you mean? Today it defaults to disabled, a patch changing > that should also change the docs? What I failed to say is that the other options document what the default it, this one doesn't yet. Christoph
On 07.11.23 17:08, Andres Freund wrote: >> make world >> make check-world >> make install-world >> >> that encompasses everything. >> >> Now with meson to handle the documentation one needs to remember a variety of additional targets. (There is a risk thatonce this gets more widespread, more people will submit broken documentation.) > install-world with meson also installs docs. Ok, I didn't know about ninja install-world. That works for me. Maybe a "world" target would also be good. I played around with this a bit and noticed some files missing or in the wrong place. See two attached patches (plus e9f075f9a1 already committed).
Attachment
On 07.11.23 17:40, Alvaro Herrera wrote: > If the problem is broken doc patches, then maybe a solution is to > include the `xmllint --noout --valid` target in whatever the check-world > equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't > seem to do that anywhere. Doing the no-output lint run is very fast > (375ms real time in my machine, whereas "make html" takes 27s). This would be a start, but it wouldn't cover everything. Lately, we require id attributes on certain elements, which is checked on the XSLT level.
Re: Peter Eisentraut > > If the problem is broken doc patches, then maybe a solution is to > > include the `xmllint --noout --valid` target in whatever the check-world > > equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't > > seem to do that anywhere. Doing the no-output lint run is very fast > > (375ms real time in my machine, whereas "make html" takes 27s). > > This would be a start, but it wouldn't cover everything. Lately, we require > id attributes on certain elements, which is checked on the XSLT level. I'd think there should be a catchy "make check-world"-equivalent that does run all reasonable check that we can tell people to run by default. Then if that takes too long, we could still offer alternatives that exclude some areas. If it's the other way round, some areas will never be checked widely. Christoph
On 08.11.23 13:55, Christoph Berg wrote: > Re: Peter Eisentraut >>> If the problem is broken doc patches, then maybe a solution is to >>> include the `xmllint --noout --valid` target in whatever the check-world >>> equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't >>> seem to do that anywhere. Doing the no-output lint run is very fast >>> (375ms real time in my machine, whereas "make html" takes 27s). >> >> This would be a start, but it wouldn't cover everything. Lately, we require >> id attributes on certain elements, which is checked on the XSLT level. > > I'd think there should be a catchy "make check-world"-equivalent that > does run all reasonable check that we can tell people to run by > default. Then if that takes too long, we could still offer > alternatives that exclude some areas. If it's the other way round, > some areas will never be checked widely. I think we could build doc/src/sgml/postgres-full.xml by default. That takes less than 0.5 seconds here and it's an intermediate target for html and man.
On 2023-Nov-08, Peter Eisentraut wrote: > I think we could build doc/src/sgml/postgres-full.xml by default. That > takes less than 0.5 seconds here and it's an intermediate target for html > and man. If that detects problems like the id attributes you mentioned, apart from the other checks in the `xmllint --noout`, then that WFM. At least with the makefile the command to produce postgres-full.xml includes --valid, so I think we're covered. -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "Having your biases confirmed independently is how scientific progress is made, and hence made our great society what it is today" (Mary Gardiner)
Looks good to me. Thanks for finding this. -- Tristan Partin Neon (https://neon.tech)
Hi, On 2023-11-08 12:04:30 +0100, Peter Eisentraut wrote: > Ok, I didn't know about ninja install-world. That works for me. Maybe a > "world" target would also be good. Yea, I thought so as well. I'll send out a patch shortly. Kinda wondering if its worth backpatching to 16. Uniformity seems useful and it's low risk. > I played around with this a bit and noticed some files missing or in the > wrong place. See two attached patches (plus e9f075f9a1 already committed). Make sense. Greetings, Andres Freund
Hi, I really like the idea of an 'help' target that prints the targets. It seemed annoying to document such targets in both the sgml docs and the input for a the help target. Particularly due to the redundancies between id attributes, the target name etc. First I generated the list of targets from within meson.build, only to later realize that that would not work when building the docs via make. So I instead added doc/src/sgml/meson-targets.txt which is lightly postprocessed for the 'help' target, and slightly more processed when building the docs. That does have some downsides, e.g. it'd be more complicated to only print targets if a relevant option is enabled. But I think it's acceptable that way. Example output: $ ninja help [0/1 1 0%] Running external command help (wrapped by meson to set env) Code Targets: all Build everything other than documentation backend Build backend and related modules bin Build frontend binaries contrib Build contrib modules pl Build procedual languages Documentation Targets: docs Build documentation in multi-page HTML format doc-html Build documentation in multi-page HTML format doc-man Build documentation in man page format doc/src/sgml/postgres-A4.pdf Build documentation in PDF format, with A4 pages doc/src/sgml/postgres-US.pdf Build documentation in PDF format, with US letter pages doc/src/sgml/postgres.html Build documentation in single-page HTML format alldocs Build documentation in all supported formats Installation Targets: install Install postgres, excluding documentation install-doc-html Install documentation in multi-page HTML format install-doc-man Install documentation in man page format install-docs Install documentation in multi-page HTML and man page formats install-quiet Like "install", but installed files are not displayed install-world Install postgres, including multi-page HTML and man page documentation uninstall Remove installed files Other Targets: clean Remove all build products test Run all enabled tests (including contrib) world Build everything, including documentation help List important targets Because of the common source, some of the descriptions in the state of this patch are a bit shorter than in the preceding commit. But I don't think that hurts much. Greetings, Andres Freund
Attachment
- v2-0001-meson-Change-default-of-selinux-feature-option-to.patch
- v2-0002-docs-Document-with-selinux-Dselinux-options-centr.patch
- v2-0003-meson-docs-Add-doc-html-man-targets.patch
- v2-0004-meson-Add-world-target.patch
- v2-0005-docs-meson-Add-documentation-for-important-build-.patch
- v2-0006-meson-Add-help-target-build-docs-from-a-common-so.patch
- v2-0007-meson-Add-Dpkglibdir-option.patch
Hi, On 2023-11-08 13:55:02 +0100, Christoph Berg wrote: > Re: Peter Eisentraut > > > If the problem is broken doc patches, then maybe a solution is to > > > include the `xmllint --noout --valid` target in whatever the check-world > > > equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't > > > seem to do that anywhere. Doing the no-output lint run is very fast > > > (375ms real time in my machine, whereas "make html" takes 27s). > > > > This would be a start, but it wouldn't cover everything. Lately, we require > > id attributes on certain elements, which is checked on the XSLT level. > > I'd think there should be a catchy "make check-world"-equivalent that > does run all reasonable check that we can tell people to run by > default. Then if that takes too long, we could still offer > alternatives that exclude some areas. If it's the other way round, > some areas will never be checked widely. The 'test' target (generated by meson, otherwise I'd have named it check), runs all enabled tests. You obviously can run a subset if you so desire. Greetings, Andres Freund
Hi, On 2023-11-08 16:19:51 +0100, Peter Eisentraut wrote: > On 08.11.23 13:55, Christoph Berg wrote: > > Re: Peter Eisentraut > > > > If the problem is broken doc patches, then maybe a solution is to > > > > include the `xmllint --noout --valid` target in whatever the check-world > > > > equivalent is for meson. Looking at doc/src/sgml/meson.build, we don't > > > > seem to do that anywhere. Doing the no-output lint run is very fast > > > > (375ms real time in my machine, whereas "make html" takes 27s). > > > > > > This would be a start, but it wouldn't cover everything. Lately, we require > > > id attributes on certain elements, which is checked on the XSLT level. > > > > I'd think there should be a catchy "make check-world"-equivalent that > > does run all reasonable check that we can tell people to run by > > default. Then if that takes too long, we could still offer > > alternatives that exclude some areas. If it's the other way round, > > some areas will never be checked widely. > > I think we could build doc/src/sgml/postgres-full.xml by default. That > takes less than 0.5 seconds here and it's an intermediate target for html > and man. That does require the docbook dtd to be installed, afaict. I think we would need a configure test for that to be present if we want to build it by default, otherwise we'll cause errors on plenty systems that don't get them today. The docbook dts aren't a huge dependency, but still. Some OSs might not have a particularly install source for them, e.g. windows. I don't think that'd detect the missing ids? Greetings, Andres Freund
On 09.11.23 01:59, Andres Freund wrote: >> I think we could build doc/src/sgml/postgres-full.xml by default. That >> takes less than 0.5 seconds here and it's an intermediate target for html >> and man. > That does require the docbook dtd to be installed, afaict. I think we would > need a configure test for that to be present if we want to build it by > default, otherwise we'll cause errors on plenty systems that don't get them > today. The docbook dts aren't a huge dependency, but still. Some OSs might > not have a particularly install source for them, e.g. windows. I was thinking we would do it only if the required tools are found. Basically like postgres_full_xml = custom_target('postgres-full.xml', input: 'postgres.sgml', output: 'postgres-full.xml', depfile: 'postgres-full.xml.d', command: [xmllint, '--nonet', '--noent', '--valid', '--path', '@OUTDIR@', '-o', '@OUTPUT@', '@INPUT@'], depends: doc_generated, - build_by_default: false, + build_by_default: xmllint_bin.found(), ) Besides giving you a quick validity check of the XML, this also builds the doc_generated, which draw from non-doc source files, so this would also serve to check that those are sound and didn't mess up the docs. > I don't think that'd detect the missing ids? Right, it wouldn't do that.
On 09.11.23 00:21, Andres Freund wrote: > Example output: This is very nice! > $ ninja help > [0/1 1 0%] Running external command help (wrapped by meson to set env) > Code Targets: > all Build everything other than documentation > backend Build backend and related modules > bin Build frontend binaries > contrib Build contrib modules > pl Build procedual languages ok > Documentation Targets: > docs Build documentation in multi-page HTML format > doc-html Build documentation in multi-page HTML format > doc-man Build documentation in man page format > doc/src/sgml/postgres-A4.pdf Build documentation in PDF format, with A4 pages > doc/src/sgml/postgres-US.pdf Build documentation in PDF format, with US letter pages > doc/src/sgml/postgres.html Build documentation in single-page HTML format > alldocs Build documentation in all supported formats > > Installation Targets: > install Install postgres, excluding documentation This should probably read "Install everything other than documentation", to mirror "all" above. (Otherwise one might think it installs just the backend.) > install-doc-html Install documentation in multi-page HTML format > install-doc-man Install documentation in man page format > install-docs Install documentation in multi-page HTML and man page formats There is a mismatch between "docs" and "install-docs". (As was previously discussed, I'm in the camp that "docs" should be html + man.) > install-quiet Like "install", but installed files are not displayed > install-world Install postgres, including multi-page HTML and man page documentation Suggest "Install everything, including documentation" (matches "world"). > uninstall Remove installed files > > Other Targets: > clean Remove all build products > test Run all enabled tests (including contrib) > world Build everything, including documentation Shouldn't that be under "Code Targets"? > help List important targets
Hi, On 2023-11-09 15:32:39 +0100, Peter Eisentraut wrote: > On 09.11.23 01:59, Andres Freund wrote: > > > I think we could build doc/src/sgml/postgres-full.xml by default. That > > > takes less than 0.5 seconds here and it's an intermediate target for html > > > and man. > > That does require the docbook dtd to be installed, afaict. I think we would > > need a configure test for that to be present if we want to build it by > > default, otherwise we'll cause errors on plenty systems that don't get them > > today. The docbook dts aren't a huge dependency, but still. Some OSs might > > not have a particularly install source for them, e.g. windows. > > I was thinking we would do it only if the required tools are found. > Basically like > > postgres_full_xml = custom_target('postgres-full.xml', > input: 'postgres.sgml', > output: 'postgres-full.xml', > depfile: 'postgres-full.xml.d', > command: [xmllint, '--nonet', '--noent', '--valid', > '--path', '@OUTDIR@', '-o', '@OUTPUT@', '@INPUT@'], > depends: doc_generated, > - build_by_default: false, > + build_by_default: xmllint_bin.found(), > ) We don't get to that point if xmllint isn't found... > Besides giving you a quick validity check of the XML, this also builds the > doc_generated, which draw from non-doc source files, so this would also > serve to check that those are sound and didn't mess up the docs. Unfortunately presence of xmllint doesn't guarantee presence of the relevant DTDs. Without docbook-xml installed, you'll get something like ../../../../../home/andres/src/postgresql/doc/src/sgml/postgres.sgml:21: warning: failed to load external entity "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" ]> ^ and a bunch of other subsequent errors. I think if we want to do this, we'd need a configure time check for being able to validate a document with <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"... That's certainly doable. If we go there, we imo also should check if the relevant xslt stylesheets are installed. Greetings, Andres Freund
On 09.11.23 18:52, Andres Freund wrote: >> Besides giving you a quick validity check of the XML, this also builds the >> doc_generated, which draw from non-doc source files, so this would also >> serve to check that those are sound and didn't mess up the docs. > Unfortunately presence of xmllint doesn't guarantee presence of the relevant > DTDs. Without docbook-xml installed, you'll get something like > > ../../../../../home/andres/src/postgresql/doc/src/sgml/postgres.sgml:21: warning: failed to load external entity"http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" > ]> > ^ > > and a bunch of other subsequent errors. > > > I think if we want to do this, we'd need a configure time check for being able > to validate a document with > <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN""http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"... We used to have exactly such a check in configure, but it was removed in 4823c4f6ac. I suppose we could look into reviving that.
Some comments on your patches: v2-0001-meson-Change-default-of-selinux-feature-option-to.patch Ok v2-0002-docs-Document-with-selinux-Dselinux-options-centr.patch Ok, but "selinux" should be "SELinux" when referring to the product. v2-0003-meson-docs-Add-doc-html-man-targets.patch We have make targets "html" and "man", so I suggest we make the meson targets the same. v2-0004-meson-Add-world-target.patch AFAICT, this world target doesn't include the man target. (Again, this would all work better if we added "man" to "docs".) v2-0005-docs-meson-Add-documentation-for-important-build-.patch It's nice to document this, but it's weird that we only document the meson targets, not the make targets. v2-0006-meson-Add-help-target-build-docs-from-a-common-so.patch Here also, making this consistent and uniform with make would be useful. v2-0007-meson-Add-Dpkglibdir-option.patch Normally, the pkgFOOdir variables are just FOOdir plus package name. I don't feel comfortable allowing those to be separately set. We don't allow that with configure; this just arose from a Debian patch. The description "location to dynamically loadable modules" is too narrow. Consider for example, another proposed patch, where we are doing some preprocessing on postgres.bki at build time. Since that makes postgres.bki platform dependent, it should really be moved from share (datadir in configure parlance) to pkglibdir. So then we have things in there that are not loadable modules. I don't know how that affects Debian packaging, but this patch might not be the right one. I suggest we leave this patch for a separate discussion.
Hi, On 2023-11-14 21:16:13 +0100, Peter Eisentraut wrote: > Some comments on your patches: > > v2-0001-meson-Change-default-of-selinux-feature-option-to.patch > > Ok > > v2-0002-docs-Document-with-selinux-Dselinux-options-centr.patch > > Ok, but "selinux" should be "SELinux" when referring to the product. Will apply with that fix. > v2-0003-meson-docs-Add-doc-html-man-targets.patch > > We have make targets "html" and "man", so I suggest we make the meson > targets the same. Hm, ok. > v2-0004-meson-Add-world-target.patch > > AFAICT, this world target doesn't include the man target. (Again, this > would all work better if we added "man" to "docs".) I agree with that sentiment - I only moved to the current arrangement after Tom argued forcefully against building both. The situation in the make world is weird: "make docs" in the toplevel builds both, because it's defined as docs: $(MAKE) -C doc all Buf if you "make -C doc/src/sgml" (or are in doc/src/sgml), we only build html, as the default target is explicitly just html: # Make "html" the default target, since that is what most people tend # to want to use. html: There's no real way of making the recursive-make and non-recursive ninja coherent. There's no equivalent to default target in a sudirectory with ninja (or non-recursive make). > v2-0005-docs-meson-Add-documentation-for-important-build-.patch > > It's nice to document this, but it's weird that we only document the meson > targets, not the make targets. I think it'd have been good if we had documented the important targets with make. But I don't think documenting them as a prerequisite to documenting the meson targets makes much sense. > v2-0006-meson-Add-help-target-build-docs-from-a-common-so.patch > > Here also, making this consistent and uniform with make would be useful. What precisely are you referring to here? Also adding a help target? Or just consistency between what the "docs" target does? > v2-0007-meson-Add-Dpkglibdir-option.patch > > Normally, the pkgFOOdir variables are just FOOdir plus package name. I > don't feel comfortable allowing those to be separately set. We don't allow > that with configure; this just arose from a Debian patch. Right - but Debian's desire seems quite sensible. The need to have multiple postgres versions installed in parallel is quite widespread. > The description "location to dynamically loadable modules" is too narrow. > Consider for example, another proposed patch, where we are doing some > preprocessing on postgres.bki at build time. Since that makes postgres.bki > platform dependent, it should really be moved from share (datadir in > configure parlance) to pkglibdir. I think I cannot be faulted for documenting the current use of the directory :). Separately, I'm not really convinced that moving some build time values into postgres.bki is useful, but that's a matter for a different thread. > So then we have things in there that are not loadable modules. I don't know > how that affects Debian packaging, but this patch might not be the right > one. I'm not really seeing why that'd affect pkglibdir being adjustable, besides needing to tweak the description of pkglibdir? > I suggest we leave this patch for a separate discussion. Fair enough. Thanks for the review, Andres Freund
Hi, On 2023-11-14 11:58:53 +0100, Peter Eisentraut wrote: > On 09.11.23 18:52, Andres Freund wrote: > > I think if we want to do this, we'd need a configure time check for being able > > to validate a document with > > <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN""http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"... > > We used to have exactly such a check in configure, but it was removed in > 4823c4f6ac. I suppose we could look into reviving that. Yea, that change was obsoleted by xmllint/xsltproc not being able to fetch the dtd over network anymore... And the performance issue 4823c4f6ac notes also doesn't apply anymore, as we use -nonet since 969509c3f2e. Greetings, Andres Freund
Hi, On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > v2-0004-meson-Add-world-target.patch > > > > AFAICT, this world target doesn't include the man target. (Again, this > > would all work better if we added "man" to "docs".) > > I agree with that sentiment - I only moved to the current arrangement after > Tom argued forcefully against building both. Another message in this thread made me realize that I actually hadn't implemented it at all - it was Tom in 969509c3f2e In HEAD, also document how to build docs using Meson, and adjust "ninja docs" to just build the HTML docs, for consistency with the default behavior of doc/src/sgml/Makefile. I think that change was just ill-advised, given that the top-level make target actually *does* build both html and man: > The situation in the make world is weird: > "make docs" in the toplevel builds both, because it's defined as > > docs: > $(MAKE) -C doc all Notwithstanding this: > Buf if you "make -C doc/src/sgml" (or are in doc/src/sgml), we only build > html, as the default target is explicitly just html: As the obvious thing for people that really just want to build html with ninja would be to just use the doc-html (to-be-renamed to "html") target. Greetings, Andres Freund
Hi, On 2023-11-14 16:30:24 -0800, Andres Freund wrote: > On 2023-11-14 16:22:31 -0800, Andres Freund wrote: > > > v2-0004-meson-Add-world-target.patch > > > > > > AFAICT, this world target doesn't include the man target. (Again, this > > > would all work better if we added "man" to "docs".) > > > > I agree with that sentiment - I only moved to the current arrangement after > > Tom argued forcefully against building both. > > Another message in this thread made me realize that I actually hadn't > implemented it at all - it was Tom in 969509c3f2e > > In HEAD, also document how to build docs using Meson, and adjust > "ninja docs" to just build the HTML docs, for consistency with the > default behavior of doc/src/sgml/Makefile. > > > I think that change was just ill-advised, given that the top-level make target > actually *does* build both html and man: > > > The situation in the make world is weird: > > "make docs" in the toplevel builds both, because it's defined as > > > > docs: > > $(MAKE) -C doc all > > Notwithstanding this: > > > Buf if you "make -C doc/src/sgml" (or are in doc/src/sgml), we only build > > html, as the default target is explicitly just html: > > As the obvious thing for people that really just want to build html with ninja > would be to just use the doc-html (to-be-renamed to "html") target. I pushed the first two commits (the selinux stuff) and worked a bit more on the subsequent ones. - As requested, I've renamed the 'doc-html' and 'doc-man' targets to just 'html' and 'man'. Which then seems to also necessitates renaming the existing install-doc-{html,man}. I'm not so sure about this change, likely because I use autocomplete to remember the spelling of ninja (or make) targets, which is easier with [install-]doc-{html,man} than with [install-]{html,man}. - I added a commit to change what 'docs' builds, undoing that part of 969509c3f2e. I also moved the 'all' target in doc/src/sgml/Makefile up to the 'html' target to make things less confusing there, as discussed in the thread referenced in the commit message. Because of the 'html' target, Tom can still just build html easily. - I renamed 'meson-targets.txt' to 'targets-meson.txt' and renamed other files to match. One reason is that meson tries to prevent conflict between its internal targets by prefixing them with 'meson-', and the old names conflicted with that rule. If we ever wanted to add something similar for make, the new naming also seems better. - I added documentation for some developer targets (reformat-dat-files, expand-dat-files, update-unicode) I didn't move 'world' in the docs, as it doesn't quite seem right in the "code targets" section? I attached the pkglibdir thing again, even though I don't plan to push it or really review it further. Thought it might still be interesting for Christoph. Greetings, Andres Freund
Attachment
On 17.11.23 19:53, Andres Freund wrote: > I pushed the first two commits (the selinux stuff) and worked a bit more on > the subsequent ones. Patches 0001 through 0004 look good to me. Some possible small tweaks in 0004: + perl, '-ne', 'next if /^#/; print', If you're going for super-brief mode, you could also use "perl -p" and drop the "print". Put at least two spaces between the "columns" in targets-meson.txt: + doc/src/sgml/postgres-A4.pdf Build documentation in PDF format, with ^^
Hi, On 2023-11-20 08:27:48 +0100, Peter Eisentraut wrote: > On 17.11.23 19:53, Andres Freund wrote: > > I pushed the first two commits (the selinux stuff) and worked a bit more on > > the subsequent ones. > > Patches 0001 through 0004 look good to me. Cool, I pushed them now. > Some possible small tweaks in 0004: > > + perl, '-ne', 'next if /^#/; print', > > If you're going for super-brief mode, you could also use "perl -p" and drop > the "print". I thought this didn't add much, so I didn't go there. > Put at least two spaces between the "columns" in targets-meson.txt: > > + doc/src/sgml/postgres-A4.pdf Build documentation in PDF format, with > ^^ I did adopt this. One remaining question is whether we should adjust install-doc-{html,man} to be install-{html,man}, to match the docs targets. Greetings, Andres Freund
On 21.11.23 02:56, Andres Freund wrote: > One remaining question is whether we should adjust install-doc-{html,man} to > be install-{html,man}, to match the docs targets. Ah didn't notice that one; yes please.
On 21.11.23 14:23, Peter Eisentraut wrote: > On 21.11.23 02:56, Andres Freund wrote: >> One remaining question is whether we should adjust >> install-doc-{html,man} to >> be install-{html,man}, to match the docs targets. > > Ah didn't notice that one; yes please. I think this was done?