Thread: Logger subprocess for win32

After many hours of investigation trying to get the logger subprocess
running in win32, I had to realize that win32 differences make it
impossible to implement it in a similar fashion. Moreover, only the
eventlog facility currently guarantees all elog messages to be logged
undisturbed, stderr does not.

After finding out which handles get inherited in which situation I
frequently found the output of the stderr catching pipe being garbled.
Especially the messages from the boot process were nearly completely
trunctated. A test program (10 childs writing every 0-100ms) showed that
redirecting stderr of several subprocesses into the same pipe will lead
to one process overwriting the other's output. This is in contrast to
Linux' behaviour. NB: this also applies if stderr is redirected on the
shell level.

Apparently, it's necessary to redirect all subprocesses stderr into
separate pipes, in order to read them atomically and write them
non-garbled to a single stderr (or file) output. I'll be implementing
this as a postmaster thread waiting on multiple pipes (one for each
client). In order to avoid redirecting and re-redirecting stderr in
postmaster every time a subprocess is launched, the pipe handled
assigned to that new process should be transferred using
read/write_backend_variables, and the new process does the actual
redirection.

Regards,
Andreas

Andreas Pflug <pgadmin@pse-consulting.de> writes:
> A test program (10 childs writing every 0-100ms) showed that
> redirecting stderr of several subprocesses into the same pipe will lead
> to one process overwriting the other's output.

Yuck.  Count on Microsoft for incompetent implementations of basic
facilities :-(

> I'll be implementing this as a postmaster thread waiting on multiple
> pipes (one for each client).

If you do, I'll reject it out of hand.  The point of having a separate
postmaster process is for it to be as bulletproof and uncrashable as
possible.  Putting such a thing as that into the postmaster will degrade
its reliability significantly.

If the multithreading were done in a separate logger subprocess, this
complaint wouldn't apply, but I was kinda hoping that we could make the
logger just as simple and high-reliability as the postmaster :-(

Are there any other alternatives?

We could apply the patch and make it #ifndef WIN32 for now, but given
the lateness of the hour I'm inclined to say that we'd best postpone
the whole thing to 7.6 ... certainly I'm running out of time to review
it before the end of the month, even if we had a finished patch now.

            regards, tom lane

PS: is it possible that the observed unreliability is not Redmond's
fault, but is something wrong with our pg_pipe code?

Tom Lane wrote:
> Yuck.  Count on Microsoft for incompetent implementations of basic
> facilities :-(

I won't contradict.

>
>>I'll be implementing this as a postmaster thread waiting on multiple
>>pipes (one for each client).
>
>
> If you do, I'll reject it out of hand.  The point of having a separate
> postmaster process is for it to be as bulletproof and uncrashable as
> possible.  Putting such a thing as that into the postmaster will degrade
> its reliability significantly.

We're not in linux. Threading is much more stable and calculatable
compared to **ix (if the software part is designed for that, I'm not
talking about complete backends), and the code would be quite short (we
already have threads in the postmaster...)
It's just a main loop scanning for data appearing in a pipe, and
forwarding into another file handle. shmem free, of course.

>
> If the multithreading were done in a separate logger subprocess, this
> complaint wouldn't apply, but I was kinda hoping that we could make the
> logger just as simple and high-reliability as the postmaster :-(

I agree, the last process to die should be the logger to catch as many
information as possible. This is the case in linux, at least the logger
subprocess will die after the postmaster (IsPostmasterAlive). We could
even invent an additional delay after postmaster exit, to catch final
pgstat output.

Hm. The problem is, if a subprocess should scan that stderr pipes, the
postmaster must know those handles as well to hand it over to the new
child. This means, it must create them inheritable. We certainly don't
want to inherit 100 pipe handles to any subprocess, if configured for
100 max clients. I'll think about that.

> Are there any other alternatives?

I tried really hard. Alternative is not to use stderr at all, and write
explicitely to individual pipes. That's really the same.


> We could apply the patch and make it #ifndef WIN32 for now,

The logger subprocess patch is already made WIN32 ready (ignoring file
logging stuff).

This thread is *not* mainly about the syslogger stuff, initially I
selected a misleading title; sorry. As I said, even standard command
line redirection of stderr is subject to M$ destruction.


> PS: is it possible that the observed unreliability is not Redmond's
> fault, but is something wrong with our pg_pipe code?


No, I have a very small test program (pg*-free) compiled with gcc or
vc6, which shows the stderr problem. pgpipe doesn't work anyway, because
it creates a pair of sockets, but M$ won't redirect stderr to sockets,
only files or pipes. I wonder why pgpipe is implemented like this, maybe
because the initial coder didn't know how to inherit handles obtained by
_pipe(...). Documentation is really not helpful on that topic. After
all, I found out by complete inspection of possibilities.

Regards,
Andreas