Thread: Images in the official documentation

Daniel Westermann <daniel.westermann@dbi-services.com> writes:
> Is there an agreement not to include that pictures for any reason?

This has been discussed repeatedly (see the archives).  We'd surely
like the ability to include figures, but we've not found any tools that
met the varying requirements people wanted to set --- mainly, that it
be possible to commit readable source text for figures into our git
repo and have updates that were reviewable (ie, didn't amount to a 100%
replacement of one set of gibberish with another one).  In the distant
past, as I recall, we had a GIF or two; but we abandoned that on the
grounds that it was unmaintainable and also incompatible with some
documentation output formats.  I'm not too sure what the state of
play is on the latter point, now that we've switched to XML.

            regards, tom lane

On 2/23/18 11:21, Tom Lane wrote:
> In the distant
> past, as I recall, we had a GIF or two; but we abandoned that on the
> grounds that it was unmaintainable and also incompatible with some
> documentation output formats.  I'm not too sure what the state of
> play is on the latter point, now that we've switched to XML.

The complications with the image formats in the past were mainly around
what ((pdf)jade)tex would accept.  The tools have shifted a bit now, and
the zoo formats is a different one.  Nothing that a few make rules
couldn't address, though, I think.

The issue of how to manage the sources is still the same, though.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut
<peter.eisentraut@2ndquadrant.com> wrote:
> On 2/23/18 11:21, Tom Lane wrote:
>> In the distant
>> past, as I recall, we had a GIF or two; but we abandoned that on the
>> grounds that it was unmaintainable and also incompatible with some
>> documentation output formats.  I'm not too sure what the state of
>> play is on the latter point, now that we've switched to XML.
>
> The complications with the image formats in the past were mainly around
> what ((pdf)jade)tex would accept.  The tools have shifted a bit now, and
> the zoo formats is a different one.  Nothing that a few make rules
> couldn't address, though, I think.
>
> The issue of how to manage the sources is still the same, though.

SVG format is ascii based vector format. We made experimental pdf with pictures
http://www.sai.msu.su/~megera/postgres/files/postgres-11-diagram.pdf
(GIN AM diagram, Appendix L).

Appendix L also demonstrates our sample database with step-by-step
introduction to Postgres for beginners.  We have a separate book for beginners,
which we released under BSD license and  it's available on
russian/english languages.
Our experience shows, that people really appreciate it. I hope we will
have time at PGCon
to discuss documentation somehow.

>
> --
> Peter Eisentraut              http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>

Oleg Bartunov <obartunov@gmail.com> writes:
> On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut
> <peter.eisentraut@2ndquadrant.com> wrote:
>> The issue of how to manage the sources is still the same, though.

> SVG format is ascii based vector format.

Yeah.  I think where the last discussion left this was that we'd be
willing to accept SVG-format figures, but we were having a hard time
figuring out what tools to recommend for editing them, because available
editors have no respect for readability or preserving small deltas when
rewriting SVG files.

            regards, tom lane

As an addition to my mail from January 2016 concerning graphics (https://www.postgresql.org/message-id/568A9148.30303%40purtz.de) I propose to use SVG (after switching to XML) - but not an SVG which is generated by Inkscape or similar tools. Those editors generate very ugly and chatty commands. This form is not easy to read or understand. Therefore we shall use nothing but a simple text editor and write every line by our self. The process is divided into two parts:

As a basis we shall develop an SVG library containing a bunch of "atomic" symbols of simple graphical elements (rectangle, arrow, ...) up to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines creating those symbols shall accept parameters for position, size, rotation, colour, ... . This library shortens the individual SVG files, it ensures a consistent rendering of common graphical elements, it is diff-able, and it will reach a stable state - some day.

The real graphics shall use the elements of the library and add individual SVG elements. The rules for this part are the same as above: create SVG commands with vi (or similar), store it in git.
If such an approach works (we must distribute the docs across a wide range of different systems, a proof-of-system is necessary) and the community accepts my proposal, I would like to work on the library-part - starting after finishing my actual project in about 6 weeks from now. The attached file contains a very first draft as of Jan. 2016.

Kind regards
Jürgen Purtz

Hi %,

I am working with PostgreSQL documentation quite a few years now and I am almost happy. What I think is completely missing (especially if you compare to commercial product documentation) are pictures that illustrate a topic, e.g. the relation of instance->database->user/role->schema->objects.

Is there an agreement not to include that pictures for any reason? I can not promise that I find time for that in the near future but if that will be appreciated I am willing to spend time on that to make the documentation even better.

Regards

Daniel

> On Feb 25, 2018, at 4:00 AM, Jürgen Purtz <juergen@purtz.de> wrote:
>
> As an addition to my mail from January 2016 concerning graphics
(https://www.postgresql.org/message-id/568A9148.30303%40purtz.de)I propose to use SVG (after switching to XML) - but
notan SVG which is generated by Inkscape or similar tools. Those editors generate very ugly and chatty commands. This
formis not easy to read or understand. Therefore we shall use nothing but a simple text editor and write every line by
ourself. The process is divided into two parts: 
> As a basis we shall develop an SVG library containing a bunch of "atomic" symbols of simple graphical elements
(rectangle,arrow, ...) up to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines
creatingthose symbols shall accept parameters for position, size, rotation, colour, ... . This library shortens the
individualSVG files, it ensures a consistent rendering of common graphical elements, it is diff-able, and it will reach
astable state - some day. 
>
> The real graphics shall use the elements of the library and add individual SVG elements. The rules for this part are
thesame as above: create SVG commands with vi (or similar), store it in git. 
> If such an approach works (we must distribute the docs across a wide range of different systems, a proof-of-system is
necessary)and the community accepts my proposal, I would like to work on the library-part - starting after finishing my
actualproject in about 6 weeks from now. The attached file contains a very first draft as of Jan. 2016. 

Writing SVG by hand maybe doesn't seem the best idea.

I understand the attraction to people who want to store everything as diffable text, but images of this sort are
unlikelyto get updated by others, which means they're unlikely to be maintained as the things they're intended to
documentchange. It also means that the people best suited to generating diagrams are the least likely to do so, and
vice-versa.

Cheers,
  Steve

> On Feb 25, 2018, at 6:45 PM, Craig Ringer <craig@2ndquadrant.com> wrote:
>
> On 26 February 2018 at 04:12, Steve Atkins <steve@blighty.com> wrote:
>
> Writing SVG by hand maybe doesn't seem the best idea.
>
> I understand the attraction to people who want to store everything as diffable text, but images of this sort are
unlikelyto get updated by others, which means they're unlikely to be maintained as the things they're intended to
documentchange. It also means that the people best suited to generating diagrams are the least likely to do so, and
vice-versa.
>
>
>
> Yeah, I think it'd just effectively preserve the status quo by rendering anyone who's willing to add images and
designsto the docs unable - or unlikely to be willing - to do so.  

Yup. I do think that graphics would be nice in a few places, and that SVG is likely the best format for them.

There are quite a few tools that could be integrated with varying amounts of effort into the documentation generation
workflow.


# ASCII language or ascii art to SVG

# asciitosvg

https://github.com/dhobsd/asciitosvg

Inspired by ditaa, similar in functionality

## blockdiag, seqdiag, actdiag, nwdiag

http://blockdiag.com

Generates various box and arrow diagrams from a DOT-ish input language.

## ditaa

It takes ascii art of box and arrow diagrams and turns them into nice svg. Also supports boxes that look like "storage"
cylinders,documents, clouds and computers. 

## erd

https://github.com/BurntSushi/erd

Entity relationship diagrams, from a plain text input. Uses DOT and graphviz under the covers.

## Markdeep

http://casual-effects.com/markdeep/

In-browser javascript rendering of ascii art, particularly boxes and arrows.

Alternate implementation, https://github.com/blampe/goat, converts to SVG via CLI.

## Mermaid

Flowcharts and sequence diagrams from a markdown-esque input.

## mscgen

Message sequence chart inputs in a DOT-ish language to SVG

## plantuml

http://plantuml.com

It supports a human-editable descriptive text input language and generates from it:

  sequence diagrams
  various box + arrow style diagrams
  flowcharts
  state diagrams
  etc.


## shaape

https://github.com/christiangoltz/shaape

Converts ascii art to SVG. Rather nice.

## svgbob

https://github.com/ivanceras/svgbobrus

Ascii art to SVG. Likely good for boxes and arrows.

## syntrax

https://kevinpt.github.io/syntrax/

Railroad diagrams (like the ones SQLite docs are known for).

## umlet

http://www.umlet.com

Pointy-clicky editor, but also converts plain text to uml, sequence, activity diagrams

# Interactive editor

## SVG-Edit

https://github.com/SVG-Edit/svgedit

Open source, browser based interactive SVG editor. Seems to generate fairly clean SVG.

Cheers,
  Steve

Craig Ringer <craig@2ndquadrant.com> writes:
> On 26 February 2018 at 04:12, Steve Atkins <steve@blighty.com> wrote:
>> Writing SVG by hand maybe doesn't seem the best idea.
>> I understand the attraction to people who want to store everything as
>> diffable text, but images of this sort are unlikely to get updated by
>> others, which means they're unlikely to be maintained as the things they're
>> intended to document change. It also means that the people best suited to
>> generating diagrams are the least likely to do so, and vice-versa.

> Yeah, I think it'd just effectively preserve the status quo by rendering
> anyone who's willing to add images and designs to the docs unable - or
> unlikely to be willing - to do so.

This is an entirely reasonable complaint.  But I don't see how we'd cope
with patches that rewrite an entire SVG file because the patch author's
WYSIWG editor emits its output in a style randomly different from the tool
the previous patch author used.  It seems like such patches would be
effectively unreviewable, and certainly incapable of being merged.

Maybe we could improve matters a bit by insisting that everyone use the
same version of the same SVG-editing tool.  But that's not too practical.
Worse, from what I've seen, even that would not really fix the problem.
The tools simply don't give a damn about comparability of their dump
files.  I don't blame their authors exactly (try diffing Postgres data
file changes :-() but that doesn't mean it isn't a problem for us.

How can we resolve these issues?

            regards, tom lane

Craig Ringer <craig@2ndquadrant.com> writes:
> On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> How can we resolve these issues?

> Question the assumptions and requirements. Why do we actually _need_
> diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
> impact if we don't have it?

Well, I'll tell you exactly why I'm being sticky about this: we've been
down this road before.  We used to have some figures in .gif format,
and one of the problems with them was they were too hard to update.
I don't buy the "they won't need updates" argument for a second, either.
For instance, I recall that one of the images we had was a diagram of
the system catalog cross-references, and it was constantly out of date
because of the difficulty of updating it.

Admittedly, this was 15+ years ago.  Maybe the state of the art in
figure editors has advanced to the point where it won't be so hard.
But color me suspicious.

            regards, tom lane

On Mon, Feb 26, 2018 at 10:23 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Craig Ringer <craig@2ndquadrant.com> writes:
>> On 26 February 2018 at 12:16, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> How can we resolve these issues?
>
>> Question the assumptions and requirements. Why do we actually _need_
>> diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
>> impact if we don't have it?
>
> Well, I'll tell you exactly why I'm being sticky about this: we've been
> down this road before.  We used to have some figures in .gif format,
> and one of the problems with them was they were too hard to update.
> I don't buy the "they won't need updates" argument for a second, either.
> For instance, I recall that one of the images we had was a diagram of
> the system catalog cross-references, and it was constantly out of date
> because of the difficulty of updating it.
>
> Admittedly, this was 15+ years ago.  Maybe the state of the art in
> figure editors has advanced to the point where it won't be so hard.
> But color me suspicious.


In case you missed, a couple of years ago we discussed this on pgcon:

Heikki's version:
https://wiki.postgresql.org/wiki/Figures_%26_Pics_in_Docs

Emre suggested to use Markdeep (BSD license),
http://casual-effects.com/markdeep/
http://www.sai.msu.su/~megera/postgres/gin-ascii-v2.md.html

It looks good for small diagrams, but will not work for complex stuff,
such as pg_catalog structure.


>
>                         regards, tom lane
>

On 2/26/18 20:02, Craig Ringer wrote:
> If I were maintaining the docs in a vacuum, I'd use graphviz for
> something like that, because it's a figure that does need regular
> updates and changes. And because
>  the list of fun things to do in my life definitely does not include
> hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
> default tool for a reason.
> 
> I'd be awfully tempted to generate the node-map part of the catalog
> relationship .dot file from a query, too.

I think graphviz would be a great fit for what we are discussing here.
Certainly more so then some-person-on-github's latest idea for how to
convert ASCII art into diagrams.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut wrote:
> On 2/26/18 20:02, Craig Ringer wrote:
> > If I were maintaining the docs in a vacuum, I'd use graphviz for
> > something like that, because it's a figure that does need regular
> > updates and changes. And because
> >  the list of fun things to do in my life definitely does not include
> > hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
> > default tool for a reason.
> > 
> > I'd be awfully tempted to generate the node-map part of the catalog
> > relationship .dot file from a query, too.
> 
> I think graphviz would be a great fit for what we are discussing here.
> Certainly more so then some-person-on-github's latest idea for how to
> convert ASCII art into diagrams.

... particularly so if said idea involves PHP, Haskell, Go, Python, or
any other language that we don't currently have as requirement in our
build chain.

GraphViz gets my vote, too.  It may not produce the most elegant
diagrams in the universe, but the source format is as good as we can
get.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Our discussion about grafics in the documentation reached to the conclusion that we shall use SVG, the importance to 'diff-ability' is rated differently, and there is no consensus about tools.

To push the issue forward I modify my original proposal to use plain svg files in a standard editor as follows:

• We define a 'Simplified SVG format' (SSVG)
• We create libraries where complex elements are predefined and can be referenced
• We write the source in ssvg-format
• A compiler (written in bison or xslt) converts ssvg-files to svg-files
• We extend the sgml-files to include the svg-files
• The ssvg and svg-files are located in a new svg directory, Makefile copies them to sgml and html directory
• A proof-of-concept is performed in 11beta2 for HTML and PDF generation.
• The ssvg-format may be XML (as used in the examples), JSON, C-style function calls

PRO:

• SVG 1.x has many restrictions and SVG 2.x does not make progress in the last years. Tools and Browsers support different ranges of the specification. The planned compiler cuts everything down to the basic language level, where a broad support is possible.
• Predefinded elements and default values reduce the ssvg file to a small and clear source file.
• You can embed original svg commands into ssvg files.
• When you use an editor and a browser in parallel, you get the visual result with few clicks.
• Everything is diff-able.
• The Makefile needs only slightly amendments: additional cp commands and some target-dependencies. We need no new tool.

CON:

• The development is done in a non-wysiwyg editor and without mouse.
• You have to count pixel.

Example:
PageLayout.ssvg: written in the new language
PageLayout.svg: the generated svg file (actually by hand, the compiler is not yet implemented)
storage.sgml: an additional paragraph to refer to the svg-file

<para>
  <mediaobject id="PageLayoutSVG">
    <imageobject role="html">
      <imagedata fileref="PageLayout.svg" format="SVG"/>
    </imageobject>
    <imageobject role="fo">
      <imagedata fileref="PageLayout.svg" format="SVG" scalefit="1" width="100%" contentdepth="100%"/>
    </imageobject>
  </mediaobject>
</para>

PageLayoutHtml.png: the HTML result
PageLayoutPdf.png: the PDF result

A second example: pgDump.svg within backup.sgml


Kind regards, Jürgen Purtz

Hello, Jürgen.

You wrote:

JP> Our discussion about grafics in the documentation reached to the 
JP> conclusion that we shall use SVG, the importance to 'diff-ability'
JP> is rated differently, and there is no consensus about tools.

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

JP> To push the issue forward I modify my original proposal to use   
JP> plain svg files in a standard editor as follows:


JP>   We define a 'Simplified SVG format' (SSVG)  We create libraries
JP> where complex elements are predefined and         can be
JP> referenced  We write the source in ssvg-format  A compiler
JP> (written in bison or xslt) converts ssvg-files to        
JP> svg-files  We extend the sgml-files to include the svg-files  The
JP> ssvg and svg-files are located in a new svg directory,        
JP> Makefile copies them to sgml and html directory  A
JP> proof-of-concept is performed in 11beta2 for HTML and PDF        
JP> generation.  The ssvg-format may be XML (as used in the examples),
JP> JSON,         C-style function calls   
JP>   

JP>        PRO:
JP>   
JP>   
JP>   SVG 1.x has many restrictions and SVG 2.x does not make        
JP> progress in the last years. Tools and Browsers support different  
JP> ranges of the specification. The planned compiler cuts        
JP> everything down to the basic language level, where a broad        
JP> support is possible.  Predefinded elements and default values
JP> reduce the ssvg file         to a small and clear source file. 
JP> You can embed original svg commands into ssvg files.  When you use
JP> an editor and a browser in parallel, you get the         visual
JP> result with few clicks.  Everything is diff-able.  The Makefile
JP> needs only slightly amendments: additional cp         commands and
JP> some target-dependencies. We need no new tool.  
JP>   
JP> CON:
JP>   
JP>   
JP>   The development is done in a non-wysiwyg editor and without    
JP> mouse.  You have to count pixel.  
JP>   

JP>        Example:
JP>        PageLayout.ssvg: written in the new language
JP>        PageLayout.svg: the generated svg file (actually by hand,
JP> the       compiler is not yet implemented)
JP>        storage.sgml: an additional paragraph to refer to the svg-file
JP>   
JP>        <para>
JP>          <mediaobject id="PageLayoutSVG">
JP>            <imageobject role="html">
JP>              <imagedata fileref="PageLayout.svg" format="SVG"/>
JP>            </imageobject>
JP>            <imageobject role="fo">
JP>              <imagedata fileref="PageLayout.svg" format="SVG"    
JP> scalefit="1" width="100%" contentdepth="100%"/>
JP>            </imageobject>
JP>          </mediaobject>
JP>        </para>
JP>   
JP>        PageLayoutHtml.png: the HTML result
JP>        PageLayoutPdf.png: the PDF result
JP>   
JP>        A second example: pgDump.svg within backup.sgml
JP>   
JP>   
JP>        Kind regards, Jürgen Purtz
JP>   
JP>   

JP>   
JP>   



-- 
With best wishes,
 Pavel                          mailto:pavel@gf.microolap.com

On 19.07.2018 14:06, Pavel Golub wrote:
> I disagree. From what I heard, GraphViz is the winner for now. I can
> give you my two cents: plantuml is another good choice

Ok, please give us an example - possibly the two previous graphics.

Kind regards, Jürgen

Hello, Jürgen.

You wrote:


JP> On 19.07.2018 14:06, Pavel Golub wrote:
>> I disagree. From what I heard, GraphViz is the winner for now. I can
>> give you my two cents: plantuml is another good choice

JP> Ok, please give us an example - possibly the two previous graphics.

Fair enough. Let's try different formats. Sorry, what exactly previous
graphics we're talking about?

JP> Kind regards, Jürgen






-- 
With best wishes,
 Pavel                          mailto:pavel@gf.microolap.com

Because I strongly support the involvement of SVG into our documentation, I welcome every activity to establish this goal. The mail to which Pavel had replied (https://www.postgresql.org/message-id/4ea1bacb-02ca-e967-31d7-d2a6db30abff%40purtz.de) contained the two graphics PageLayout.svg "Page Layout" and pgDump.svg "pg_dump, psql, pg_restore". Actually they are not part of our documentation. They are only part of a proof-of-concept to generate SVG and include the result into HTML and PDF output. The two examples can be used as references for comparisons with any tool.
Kind regards, Jürgen

Hello, Jürgen.

You wrote:


JP> On 19.07.2018 14:06, Pavel Golub wrote:

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

JP> Ok, please give us an example - possibly the two previous graphics.

Fair enough. Let's try different formats. Sorry, what exactly previous
graphics we're talking about?

JP> Kind regards, Jürgen

After many mails, F2F-discussions and some experiments - everything 
without a concrete result - I hereby start a new initiative concerning 
SVG tools. The attached PDF file describes my recommendation. The two 
SVG files are examples for working in this sense.

Kind regards

Jürgen Purtz

Hi Jürgen,

On 11/14/18 8:36 AM, Jürgen Purtz wrote:
> After many mails, F2F-discussions and some experiments - everything
> without a concrete result - I hereby start a new initiative concerning
> SVG tools. The attached PDF file describes my recommendation. The two
> SVG files are examples for working in this sense.

It looks like you put a lot of effort into this, thanks! Those SVGs do
look great, and it's IMO it's a much needed feature to support images in
our documentation :) I will leave it to others to discuss if the
generated SVGs are a technical fit for the documentation code.

I would recommend that you transfer the contents of your proposal to the
PostgreSQL wiki (https://wiki.postgresql.org/) as it would make it
easier for others to collaborate on and contribute to it. If you need
help with it, please feel free to ping me.

The one thing that I see is if we would like to have a style guide,
primarily around colors, for images we put into the repo. I know that
somehow correlates with the definition of "bike shed" but it could also
help reduce that in the future. As we do have a PostgreSQL color
palette, I also see that as an opportunity to reduce potential
bikeshedding as well.

Thanks!

Jonathan