Thread: proposal - psql - use pager for \watch command

Pavel Stehule <pavel.stehule@gmail.com> writes:
> I propose new psql environment variable PSQL_WATCH_PAGER. When this
> variable is not empty, then \watch command starts specified pager, and
> redirect output to related pipe. When pipe is closed - by pager, then
> \watch cycle is leaved.

I dunno, this just seems really strange.  With any normal pager,
you'd get completely unusable behavior (per the comments that you
didn't bother to change).  Also, how would the pager know where
the boundaries between successive query outputs are?  If it does
not know, seems like that's another decrement in usability.

            regards, tom lane

On Fri, Jan 8, 2021 at 10:36 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> ne 19. 4. 2020 v 19:27 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>> last week I finished pspg 3.0 https://github.com/okbob/pspg . pspg now supports pipes, named pipes very well. Today
thepspg can be used as pager for output of \watch command. Sure, psql needs attached patch. 
>>
>> I propose new psql environment variable PSQL_WATCH_PAGER. When this variable is not empty, then \watch command
startsspecified pager, and redirect output to related pipe. When pipe is closed - by pager, then \watch cycle is
leaved.
>>
>> If you want to test proposed feature, you need a pspg with cb4114f98318344d162a84b895a3b7f8badec241 commit.
>>
>> Then you can set your env
>>
>> export PSQL_WATCH_PAGER="pspg --stream"
>> psql
>>
>> SELECT * FROM pg_stat_database;
>> \watch 1
>>
>> Comments, notes?

I tried this out with pspg 4.1 from my package manager.  It seems
really useful, especially for demos.  I like it!

         * Set up rendering options, in particular, disable the pager, because
         * nobody wants to be prompted while watching the output of 'watch'.
         */
-       myopt.topt.pager = 0;
+       if (!pagerpipe)
+               myopt.topt.pager = 0;

Obsolete comment.

+static bool sigpipe_received = false;

This should be "static volatile sig_atomic_t", and I suppose our
convention name for that variable would be got_SIGPIPE.  Would it be
possible to ignore SIGPIPE instead, and then rely on another way of
knowing that the pager has quit?  But... hmm:

-                       long            s = Min(i, 1000L);
+                       long            s = Min(i, pagerpipe ? 100L : 1000L);

I haven't studied this (preexisting) polling loop, but I don't like
it.  I understand that it's there because on some systems, pg_usleep()
won't wake up for SIGINT (^C), but now it's being used for a secondary
purpose, that I haven't fully understood.  After I quit pspg (by
pressing q) while running \watch 10, I have to wait until the end of a
10 second cycle before it tries to write to the pipe again, unless I
also press ^C.  I feel like it has to be possible to achieve "precise"
behaviour somehow when you quit; maybe something like waiting for
readiness on the pager's stderr, or something like that -- I haven't
thought hard about this and I admit that I have no idea how this works
on Windows.

Sometimes I see a message like this after I quit pspg:

postgres=# \watch 10
input stream was closed

On Thu, Mar 4, 2021 at 11:28 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> čt 4. 3. 2021 v 7:37 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>> Here is a little bit updated patch - detection of end of any child process cannot be used on WIN32.

Yeah, it's OK for me if this feature only works on Unix until the
right person for the job shows up with a patch.  If there is no pspg
on Windows, how would we even know if it works?

> second version - after some thinking, I think the pager for \watch command should be controlled by option "pager"
too. When the pager is disabled on psql level, then the pager will not be used for \watch too. 

Makes sense.

+            long        s = Min(i, 100L);
+
+            pg_usleep(1000L * s);
+
+            /*
+             * in this moment an pager process can be only one child of
+             * psql process. There cannot be other processes. So we can
+             * detect end of any child process for fast detection of
+             * pager process.
+             *
+             * This simple detection doesn't work on WIN32, because we
+             * don't know handle of process created by _popen function.
+             * Own implementation of _popen function based on CreateProcess
+             * looks like overkill in this moment.
+             */
+            if (pagerpipe)
+            {
+
+                int        status;
+                pid_t    pid;
+
+                pid = waitpid(-1, &status, WNOHANG);
+                if (pid)
+                    break;
+            }
+
+#endif
+
             if (cancel_pressed)
                 break;

I thought a bit about what we're really trying to achieve here.  We
want to go to sleep until someone presses ^C, the pager exits, or a
certain time is reached.  Here, we're waking up 10 times per second to
check for exited child processes.  It works, but it does not spark
joy.

I thought about treating SIGCHLD the same way as we treat SIGINT: it
could use the siglongjmp() trick for a non-local exit from the signal
handler.  (Hmm... I wonder why that pre-existing code bothers to check
cancel_pressed, considering it is running with
sigint_interrupt_enabled = true so it won't even set the flag.)  It
feels clever, but then you'd still have the repeating short
pg_usleep() calls, for reasons described by commit 8c1a71d36f5.  I do
not like sleep/poll loops.  Think of the polar bears.  I need to fix
all of these, as a carbon emission offset for cfbot.

Although there are probably several ways to do this efficiently, my
first thought was: let's try sigtimedwait()!  If you block the signals
first, you have a race-free way to wait for SIGINT (^C), SIGCHLD
(pager exited) or a timeout you can specify.  I coded that up and it
worked pretty nicely, but then I tried it on macOS too and it didn't
compile -- Apple didn't implement that.  Blah.

Next I tried sigwait().  That's already used in our tree, so it should
be OK.  At first I thought that using SIGALRM to wake it up would be a
bit too ugly and I was going to give up, but then I realised that an
interval timer (one that automatically fires every X seconds) is
exactly what we want here, and we can set it up just once at the start
of do_watch() and cancel it at the end of do_watch().  With the
attached patch you get exactly one sigwait() syscall of the correct
duration per \watch cycle.

Thoughts?  I put my changes into a separate patch for clarity, but
they need some more tidying up.

I'll look at the documentation next.

On Sun, Mar 21, 2021 at 11:44 AM Thomas Munro <thomas.munro@gmail.com> wrote:
> [review]

Oh, just BTW, to save confusion for others who might try this:  It
seems there is something wrong with pspg --stream on macOS, at least
when using MacPorts.  I assumed it might be just pspg 3.1.5 being too
old (that's what MacPorts has currently), so I didn't mention it
before, but I just built pspg from your github master branch and it
has the same symptom.  It doesn't seem to repaint the screen until you
press a key.  I can see that psql is doing its job, but pspg is
sitting in select() reached from ncurses wgetch():

  * frame #0: 0x000000019b4af0e8 libsystem_kernel.dylib`__select + 8
    frame #1: 0x0000000100ca0620 libncurses.6.dylib`_nc_timed_wait + 332
    frame #2: 0x0000000100c85444 libncurses.6.dylib`_nc_wgetch + 296
    frame #3: 0x0000000100c85b24 libncurses.6.dylib`wgetch + 52
    frame #4: 0x0000000100a815e4 pspg`get_event + 624
    frame #5: 0x0000000100a7899c pspg`main + 9640
    frame #6: 0x000000019b4f9f34 libdyld.dylib`start + 4

That's using MacPorts' libncurses.  I couldn't get it to build against
Apple's libcurses (some missing functions).  It's the same for both
your V2 and the fixup I posted.  When you press a key, it suddenly
catches up and repaints all the \watch updates that were buffered.

It works fine on Linux and FreeBSD though (I tried pspg 4.1.0 from
Debian's package manager, and pspg 4.3.1 from FreeBSD's).

On Sun, Mar 21, 2021 at 10:38 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> Can somebody help me (with access on macos0 with debugging this issue?

I'll try to figure it out, but maybe after the code freeze.  I started
my programming career writing curses software a million years ago on a
couple of extinct Unixes... I might even be able to remember how it
works.  This is not a problem for committing the PSQL_WATCH_PAGER
patch, I just mentioned it here because I thought that others might
try it out on a Mac and be confused.

On Mon, Mar 22, 2021 at 5:10 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> probably there will not be an issue inside ncurses - the most complex part of get_event is polling of input sources -
ttyand some other. The pspg should not to stop there on tty reading.
 

The problem is that Apple's /dev/tty device is defective, and doesn't
work in poll().  It always returns immediately with revents=POLLNVAL,
but pspg assumes that data is ready and tries to read the keyboard and
then blocks until I press a key.  This seems to fix it:

+#ifndef __APPLE__
+               /* macOS can't use poll() on /dev/tty */
                state.tty = fopen("/dev/tty", "r+");
+#endif
                if (!state.tty)
                        state.tty = fopen(ttyname(fileno(stdout)), "r");

A minor problem is that on macOS, _GNU_SOURCE doesn't seem to imply
NCURSES_WIDECHAR, so I suspect Unicode will be broken unless you
manually add -DNCURSES_WIDECHAR=1, though I didn't check.

On Tue, Mar 23, 2021 at 1:53 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> po 22. 3. 2021 v 13:13 odesílatel Thomas Munro <thomas.munro@gmail.com> napsal:
>> The problem is that Apple's /dev/tty device is defective, and doesn't
>> work in poll().  It always returns immediately with revents=POLLNVAL,
>> but pspg assumes that data is ready and tries to read the keyboard and
>> then blocks until I press a key.  This seems to fix it:
>>
>> +#ifndef __APPLE__
>> +               /* macOS can't use poll() on /dev/tty */
>>                 state.tty = fopen("/dev/tty", "r+");
>> +#endif
>>                 if (!state.tty)
>>                         state.tty = fopen(ttyname(fileno(stdout)), "r");
>
>
> it is hell.

Heh.  I've recently spent many, many hours trying to make AIO work on
macOS, and nothing surprises me anymore.  BTW I found something from
years ago on the 'net that fits with my observation about /dev/tty:

https://www.mail-archive.com/bug-gnulib@gnu.org/msg00296.html

Curious, which other OS did you put that fallback case in for?  I'm a
little confused about why it works, so I'm not sure if it's the best
possible change, but I'm not planning to dig any further now, too many
patches, not enough time :-)

> Please, can you verify this fix?

It works perfectly for me on a macOS 11.2 system with that change,
repainting the screen exactly when it should.  I'm happy about that
because (1) it means I can confirm that the proposed change to psql is
working correctly on the 3 Unixes I have access to, and (2) I am sure
that a lot of Mac users will appreciate being able to use super-duper
\watch mode when this ships (a high percentage of PostgreSQL users I
know use a Mac as their client machine).

>> A minor problem is that on macOS, _GNU_SOURCE doesn't seem to imply
>> NCURSES_WIDECHAR, so I suspect Unicode will be broken unless you
>> manually add -DNCURSES_WIDECHAR=1, though I didn't check.
>
> It is possible -
>
> can you run "pspg --version"

Looks like I misunderstood: it is showing "with wide char support",
it's just that the "num" is 0 rather than 1.  I'm not planning to
investigate that any further now, but I checked that it can show the
output of SELECT 'špeĉiäl chârãçtérs' correctly.

On Sun, Mar 21, 2021 at 11:43 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> so 20. 3. 2021 v 23:45 odesílatel Thomas Munro <thomas.munro@gmail.com> napsal:
>> Thoughts?  I put my changes into a separate patch for clarity, but
>> they need some more tidying up.
>
> yes, your solution is much better.

Hmm, there was a problem with it though: it blocked ^C while running
the query, which is bad.  I fixed that.   I did some polishing of the
code and some editing on the documentation and comments.  I disabled
the feature completely on Windows, because it seems unlikely that
we'll be able to know if it even works, in this cycle.

-       output = PageOutput(158, pager ? &(pset.popt.topt) : NULL);
+       output = PageOutput(160, pager ? &(pset.popt.topt) : NULL);

What is that change for?

Here's a rebase, due to a conflict with 3a513067 "psql: Show all query
results by default" which moved a few things around making it harder
to use the pager for the right scope.  Lacking time, I came up with
this change to PSQLexecWatch():

+       if (printQueryFout)
+       {
+               restoreQueryFout = pset.queryFout;
+               pset.queryFout = printQueryFout;
+       }
+
        SetCancelConn(pset.db);
        res = SendQueryAndProcessResults(query, &elapsed_msec, true);
        ResetCancelConn();

        fflush(pset.queryFout);

+       if (restoreQueryFout)
+               pset.queryFout = restoreQueryFout;
+

If someone has a tidier way to factor this, I'm keen to hear it.  I'd
like to push this today.

On Wed, Apr 21, 2021 at 6:33 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> here is an rebase of Thomas's implementation

Thanks.  I finished up not committing that one for 14 because I wasn't
sure about the way to rebase it on top of 3a513067 (now reverted);
that "restore" stuff seemed a bit weird.  Let's try again in v15 CF1!

On Wed, May 12, 2021 at 5:45 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>
>
>
> st 12. 5. 2021 v 12:25 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>
>> Hi
>>
>> st 21. 4. 2021 v 8:52 odesílatel Pavel Stehule <pavel.stehule@gmail.com> napsal:
>>>
>>>
>>>
>>> st 21. 4. 2021 v 8:49 odesílatel Thomas Munro <thomas.munro@gmail.com> napsal:
>>>>
>>>> On Wed, Apr 21, 2021 at 6:33 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>>>> > here is an rebase of Thomas's implementation
>>>>
>>>> Thanks.  I finished up not committing that one for 14 because I wasn't
>>>> sure about the way to rebase it on top of 3a513067 (now reverted);
>>>> that "restore" stuff seemed a bit weird.  Let's try again in v15 CF1!
>>>
>>>
>>> Understand. Thank you
>>
>>
>> rebase
>
>
> looks so with your patch psql doesn't work on ms
>
> https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.134648

I am changing the status to "Waiting on Author" as Pavel's comments
are not addressed.

Regards,
Vignesh

On Sun, Jul 11, 2021 at 1:18 AM vignesh C <vignesh21@gmail.com> wrote:
> On Wed, May 12, 2021 at 5:45 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > looks so with your patch psql doesn't work on ms

Here's a fix for Windows.  The pqsignal() calls are #ifdef'd out.  I
also removed a few lines that were added after commit 3a513067 but
aren't needed anymore after fae65629.

On Mon, Jul 12, 2021 at 4:29 AM Thomas Munro <thomas.munro@gmail.com> wrote:
>
> On Sun, Jul 11, 2021 at 1:18 AM vignesh C <vignesh21@gmail.com> wrote:
> > On Wed, May 12, 2021 at 5:45 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> > > looks so with your patch psql doesn't work on ms
>
> Here's a fix for Windows.  The pqsignal() calls are #ifdef'd out.  I
> also removed a few lines that were added after commit 3a513067 but
> aren't needed anymore after fae65629.

Thanks for fixing the comments, CFbot also passes for the same. I have
changed the status back to "Ready for Committer".

Regards,
Vignesh

On Tue, Jul 13, 2021 at 8:20 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> po 12. 7. 2021 v 18:12 odesílatel vignesh C <vignesh21@gmail.com> napsal:
>> Thanks for fixing the comments, CFbot also passes for the same. I have
>> changed the status back to "Ready for Committer".
>
> I tested this version with the last release and with a developing version of pspg, and it works very well.

Pushed, after retesting on macOS (with the fixed pspg that has by now
arrived in MacPorts), FreeBSD and Linux.  Thanks!  I'm using this to
monitor system views when demoing new features in development, it's
nice.  Of course, I don't like the default theme, it's a bit too
MS-DOS/Norton for my taste, but the quieter themes are good :-)

Thomas Munro <thomas.munro@gmail.com> writes:
> Pushed, after retesting on macOS (with the fixed pspg that has by now
> arrived in MacPorts), FreeBSD and Linux.  Thanks!

After playing with this along the way to fixing the sigwait issues,
I have a gripe/suggestion.  If I hit control-C while the thing
is waiting between queries, eg

regression=# select now() \watch
Tue Jul 13 13:44:44 2021 (every 2s)

              now              
-------------------------------
 2021-07-13 13:44:44.396565-04
(1 row)

Tue Jul 13 13:44:46 2021 (every 2s)

              now              
-------------------------------
 2021-07-13 13:44:46.396572-04
(1 row)

^Cregression=# 

then as you can see I get nothing but the "^C" echo before the next
psql prompt.  The problem with this is that now libreadline is
misinformed about the cursor position, messing up any editing I
might try to do on the next line of input.  So I think it would
be a good idea to have some explicit final output when the \watch
command terminates, along the line of

...
Tue Jul 13 13:44:46 2021 (every 2s)

              now              
-------------------------------
 2021-07-13 13:44:46.396572-04
(1 row)

^C\watch cancelled
regression=# 

This strikes me as a usability improvement even without the
readline-confusion angle.

            regards, tom lane

On Wed, Jul 14, 2021 at 6:06 AM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> út 13. 7. 2021 v 19:50 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal:
>> After playing with this along the way to fixing the sigwait issues,
>> I have a gripe/suggestion.  If I hit control-C while the thing
>> is waiting between queries, eg
>>
>> regression=# select now() \watch
>> Tue Jul 13 13:44:44 2021 (every 2s)
>>
>>               now
>> -------------------------------
>>  2021-07-13 13:44:44.396565-04
>> (1 row)
>>
>> Tue Jul 13 13:44:46 2021 (every 2s)
>>
>>               now
>> -------------------------------
>>  2021-07-13 13:44:46.396572-04
>> (1 row)
>>
>> ^Cregression=#
>>
>> then as you can see I get nothing but the "^C" echo before the next
>> psql prompt.  The problem with this is that now libreadline is
>> misinformed about the cursor position, messing up any editing I
>> might try to do on the next line of input.  So I think it would
>> be a good idea to have some explicit final output when the \watch
>> command terminates, along the line of
>>
>> ...
>> Tue Jul 13 13:44:46 2021 (every 2s)
>>
>>               now
>> -------------------------------
>>  2021-07-13 13:44:46.396572-04
>> (1 row)
>>
>> ^C\watch cancelled
>> regression=#
>>
>> This strikes me as a usability improvement even without the
>> readline-confusion angle.
>
> I'll look at this issue.

Hi Pavel,

Do you have a patch for this?

On Mon, May 9, 2022 at 7:07 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
> út 13. 7. 2021 v 19:50 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal:
>> ^Cregression=#
>>
>> then as you can see I get nothing but the "^C" echo before the next
>> psql prompt.  The problem with this is that now libreadline is
>> misinformed about the cursor position, messing up any editing I
>> might try to do on the next line of input.  So I think it would
>> be a good idea to have some explicit final output when the \watch
>> command terminates, along the line of
>>
>> ...
>> Tue Jul 13 13:44:46 2021 (every 2s)
>>
>>               now
>> -------------------------------
>>  2021-07-13 13:44:46.396572-04
>> (1 row)
>>
>> ^C\watch cancelled
>> regression=#
>>
>> This strikes me as a usability improvement even without the
>> readline-confusion angle.
>
> here is an patch

I played with this.  On a libedit build (tested on my Mac), an easy
way to see corruption is to run eg SELECT;, then \watch 1, then ^C,
then up-arrow to see the previous command clobber the wrong columns.
On a libreadline build (tested on my Debian box), that simple test
doesn't fail in the same way.  Though there may be some other way to
make it misbehave that would take me longer to find, it's enough for
me that libedit is befuddled by what we're doing.

Do we really need the extra text?  What about just \n, so you get:

postgres=# \watch 1
...blah blah...
^C
postgres=#

This affects all release branches too.  Should we bother to fix this
there?  For them, I think the fix is just:

diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index d1ee795cb6..3a88d5d6c4 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -4992,6 +4992,9 @@ do_watch(PQExpBuffer query_buf, double sleep)
        sigint_interrupt_enabled = false;
    }

+   fprintf(pset.queryFout, "\n");
+   fflush(pset.queryFout);
+
    pg_free(title);
    return (res >= 0);
 }

Thomas Munro <thomas.munro@gmail.com> writes:
> On Mon, May 9, 2022 at 7:07 PM Pavel Stehule <pavel.stehule@gmail.com> wrote:
>> út 13. 7. 2021 v 19:50 odesílatel Tom Lane <tgl@sss.pgh.pa.us> napsal:
>>> ^C\watch cancelled
>>> regression=#

> Do we really need the extra text?  What about just \n, so you get:

> postgres=# \watch 1
> ...blah blah...
> ^C
> postgres=#

Fine by me.

> This affects all release branches too.  Should we bother to fix this
> there?  For them, I think the fix is just:

If we're doing something as nonintrusive as just adding a newline,
it'd probably be OK to backpatch.

The code needs a comment about why it's emitting a newline, though.
In particular, it had better explain why that should be conditional
on !pagerpipe, because that makes no sense to me.

            regards, tom lane

On Tue, Jun 7, 2022 at 3:23 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> The code needs a comment about why it's emitting a newline, though.
> In particular, it had better explain why that should be conditional
> on !pagerpipe, because that makes no sense to me.

Yeah.  OK, here's my take:

+               /*
+                * If the terminal driver echoed "^C",
libedit/libreadline might be
+                * confused about the cursor position.  Therefore,
inject a newline
+                * before the next prompt is displayed.  We only do
this when not using
+                * a pager, because pagers are expected to restore the
screen to a sane
+                * state on exit.
+                */

AFAIK pagers conventionally use something like termcap ti/te[1] to
restore the screen, or equivalents in tinfo etc (likely via curses).
If we were to inject an extra newline we'd just have a blank line for
nothing.  I suppose there could be a hypothetical pager that doesn't
follow that convention, and in fact both less and pspg have a -X
option to preserve last output, but in any case I expect that pagers
disable echoing, so I don't think the ^C will make it to the screen,
and furthermore ^C isn't used for exit anyway.  Rather than speculate
about the precise details, I just said "... sane state on exit".
Pavel, do you agree?

Here's how it looks after I enter and then exit Pavel's streaming pager:

$ PSQL_WATCH_PAGER='pspg --stream' ~/install/bin/psql postgres
psql (15beta1)
Type "help" for help.

postgres=# select;
--
(1 row)

postgres=# \watch 1
postgres=#

FWIW it's the same with PSQL_WATCH_PAGER='less'.

[1] https://www.gnu.org/software/termutils/manual/termcap-1.3/html_node/termcap_39.html