Thread: Speaking of pgstats
While we're talking about pgstats... There was some talk a while back about the whole bufferer/collector combination perhaps being unnecessary as well, and that it might be a good idea to simplify it down to just a collector. I'm not 100% sure what the end result of that discussion was, thouhg, and I can't find it in the archives :-( Anyway. I think this might help some of the win32 specific issues. Considering we had a lot of problems getting it up and running, most related to the "socket inheritance across two fork/exec steps", I still think there might be problems lurking there that would simply go away in a case like this. The overhead is also definitly larger on win32, considering a process taskswitch is much more expensive, and considering that we emulate the pipe using TCP... So I'd be interested in giving this a shot, but before starting I'd like to know if people think it's a worthwhile thing, or if it's likely to be rejected out-of-hand. (Of course, it can always be rejected on ipmlementation details, or on the fact that it wasn't a good idea, but if it's already known that it's not a good idea I don't want to spend time on it..) The general idea would be to still use UDP backend->stats but get rid of the pipe part (emulated by standard tcp sockets on win32), so we'd still have the "lose packets instead of blocking when falling behind". //Magnus
"Magnus Hagander" <mha@sollentuna.net> writes:
> While we're talking about pgstats... There was some talk a while back
> about the whole bufferer/collector combination perhaps being unnecessary
> as well, and that it might be a good idea to simplify it down to just a
> collector. I'm not 100% sure what the end result of that discussion was,
> thouhg, and I can't find it in the archives :-(
Yeah, I was thinking that same thing this morning. AFAIR we designed
the current structure "on paper" in a pghackers thread, and never did
any serious experimentation to prove that it was worth having the extra
process. I concur it's worth at least testing the simpler method.
> The general idea would be to still use UDP backend->stats but get rid of
> the pipe part (emulated by standard tcp sockets on win32), so we'd still
> have the "lose packets instead of blocking when falling behind".
Right.
regards, tom lane
>> The general idea would be to still use UDP backend->stats but get rid >> of >> the pipe part (emulated by standard tcp sockets on win32), so we'd >> still >> have the "lose packets instead of blocking when falling behind". > > Right. Please correct me if I am wrong, but using UDP logging on the same computer is a red herring. Any non-blocking I/O would do, no? If the buffer is full, then the non-blocking I/O send function will fail and the message is skipped. Has anyone observed UDP ever drop *written* packets on loopback? Looking at the Darwin 8 sources, it appears that the loopback streams all converge to the same stream code, which makes sense... If a kernel is too busy to handle I/O, doesn't it have higher priorities than switching to a user context?
Agent M <agentm@themactionfaction.com> writes:
> Please correct me if I am wrong, but using UDP logging on the same
> computer is a red herring. Any non-blocking I/O would do, no? If the
> buffer is full, then the non-blocking I/O send function will fail and
> the message is skipped.
Uh, not entirely. We'd like the thing to drop complete messages, not
inject partial messages into the channel causing reader parsing errors.
This is one reason for liking UDP semantics better than pipe semantics.
regards, tom lane
"Magnus Hagander" <mha@sollentuna.net> writes: > While we're talking about pgstats... There was some talk a while back > about the whole bufferer/collector combination perhaps being unnecessary > as well, and that it might be a good idea to simplify it down to just a > collector. I'm not 100% sure what the end result of that discussion was, > thouhg, and I can't find it in the archives :-( After a bit of archives-digging, I think you must be remembering this thread: http://archives.postgresql.org/pgsql-hackers/2006-01/msg00074.php which was considering not only abandoning the intermediate buffer process, but abandoning the assumption that it's OK to drop messages under load. We might or might be ready to go that far, but it's worth re-reading and reflecting --- see particularly Jan's comment at http://archives.postgresql.org/pgsql-hackers/2006-01/msg00088.php regards, tom lane
Tom Lane wrote: > "Magnus Hagander" <mha@sollentuna.net> writes: > > While we're talking about pgstats... There was some talk a while back > > about the whole bufferer/collector combination perhaps being unnecessary > > as well, and that it might be a good idea to simplify it down to just a > > collector. I'm not 100% sure what the end result of that discussion was, > > thouhg, and I can't find it in the archives :-( > > Yeah, I was thinking that same thing this morning. AFAIR we designed > the current structure "on paper" in a pghackers thread, and never did > any serious experimentation to prove that it was worth having the extra > process. I concur it's worth at least testing the simpler method. My research is in the hold queue: http://momjian.postgresql.org/cgi-bin/pgpatches_hold Subject is "Stats collector performance improvement". I am waiting for someone to confirm my tests on other platforms before moving forward, but we really should do something for 8.2. If someone else wants to work on it, go ahead. All my work is in those emails. -- Bruce Momjian http://candle.pha.pa.us EnterpriseDB http://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +