Re: COPYable logs status - Mailing list pgsql-hackers

Andrew Dunstan wrote:>> The CSVlog pipe is a separate pipe from the stderr pipe. Anything that> goes to stderr now will
continueto go to stderr, wherever that is.>> I like this scheme for a couple of reasons:> . it will include the ability
totell the real end of a message> . it will let us handle non-protocol messages (although there shouldn't> be any in
theCSVlog pipe).
 

Another important reason I went for two seperate pipes is that, in 
Windows, the pipe calls being blocking calls, the performance really 
deteriorates unless we increase the allocated buffer to the pipes 
dramatically.

On a rather decent machine, simply running the regression tests would 
consume a lot of resources, especially when it comes to the errors tests.

Rgds,
Arul Shaji


Andrew Dunstan wrote:
> 
> 
> Tom Lane wrote:
>> Alvaro Herrera <alvherre@commandprompt.com> writes:
>>  
>>> The idea of one pipe per process is not really workable, because it
>>> would mean having as many pipes as backends which does not sound very
>>> good.  But how about a mixed approach -- like have the all the backends
>>> share a pipe, controlled by an LWLock, and the auxiliary process have a
>>> separate pipe each?
>>>     
>>
>> Multiple pipes seem like a mess, and in any case the above still doesn't
>> work for stderr output produced by non-cooperative software (dynamic
>> loader for instance).
>>
>> The only solution that I can see is to invent some sort of simple
>> protocol for the syslogger pipe.  Assume that the kernel honors PIPE_BUF
>> (this assumption may need proving, see other message).  We could imagine
>> having elog.c divvy up its writes to the pipe into chunks of less than
>> PIPE_BUF bytes, where each chunk carries info sufficient to let it be
>> reassembled.  Perhaps something on the order of
>>
>>     \0 \0 2-byte-length source-PID end-flag text...
>>
>> The syslogger reassembles these by joining messages with the same
>> origination PID, until it gets one with the end-flag set.  It would need
>> enough code to track multiple in-progress messages.
>>
>> The logger would have to also be able to deal with random text coming
>> down the pipe (due to aforesaid non-cooperative software).  I would be
>> inclined to say just take any text not preceded by \0\0 as a standalone
>> message, up to the next \0\0.  Long chunks of non-protocol text would
>> risk getting treated as multiple messages, but there's probably not a
>> lot of harm in that.
>>
>> BTW, exactly what is the COPYable-logs code going to do with random
>> text?  I trust the answer is not "throw it away".
>>
>>            
>>   
> 
> The CSVlog pipe is a separate pipe from the stderr pipe. Anything that 
> goes to stderr now will continue to go to stderr, wherever that is.
> 
> I like this scheme for a couple of reasons:
> . it will include the ability to tell the real end of a message
> . it will let us handle non-protocol messages (although there shouldn't 
> be any in the CSVlog pipe).
> 
> I'll try to get a patch out for just the stderr case, which should be 
> back-patchable, then adjust the CSVlog patch to use it.
> 
> I'm thinking of handling the partial lines with a small dynahash of 
> StringInfo buffers, which get discarded whenever we don't have a partial 
> line for the PID.
> 
> cheers
> 
> andrew
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to majordomo@postgresql.org so that your
>       message can get through to the mailing list cleanly
> 
>