Thread: Re: [pgsql-hackers-win32] scalability issues on win32

> > This was an intersting Win32/linux comparison. I expected
> > Linux to scale better, but I was surprised how poorly XP
> > scaled.  It reinforces our perception that Win32 is for low
> > traffic servers.
>
> That's a bit harsh given the lack of any further investigation so far
> isn't it? Win32 can run perfectly well with other DBMSs with hundreds
of
> users.
>
> Any chance you can profile your test runs Merlin?
>

Ok, I am starting to strongly suspect the statistics collector of
various kinds of malfeasance.  Right now I am running with the stats
collector off and I am getting much better scalability and much more
deterministic query times...that is, even when under moderate to heavy
load query running times are proportional to the number of users on the
system...the system is purring like a kitten right now with over a 100
users logged in.

This coupled with the fact that I was getting random restarts with the
collector process makes me think that there is some kind of issue with
the ipc between the collector and the backends that is blocking and/or
is being improperly handled after failure.

I was running with statement level stats on under scenarios with
extremely high levels of query activity where the server might be
processing 500-1500 queries/second or more spread out over multiple
backends.

I'll look into this issue more over next several days.  I'll dip back
into the code and see if I can come up with a better
answer...unfortunately I can't run the stats collector on until I can
schedule another load test.

Merlin

Merlin Moncure wrote:
> > > This was an intersting Win32/linux comparison. I expected
> > > Linux to scale better, but I was surprised how poorly XP
> > > scaled.  It reinforces our perception that Win32 is for low
> > > traffic servers.
> >
> > That's a bit harsh given the lack of any further investigation so far
> > isn't it? Win32 can run perfectly well with other DBMSs with hundreds
> of
> > users.
> >
> > Any chance you can profile your test runs Merlin?
> >
>
> Ok, I am starting to strongly suspect the statistics collector of
> various kinds of malfeasance.  Right now I am running with the stats
> collector off and I am getting much better scalability and much more
> deterministic query times...that is, even when under moderate to heavy
> load query running times are proportional to the number of users on the
> system...the system is purring like a kitten right now with over a 100
> users logged in.
>
> This coupled with the fact that I was getting random restarts with the
> collector process makes me think that there is some kind of issue with
> the ipc between the collector and the backends that is blocking and/or
> is being improperly handled after failure.
>
> I was running with statement level stats on under scenarios with
> extremely high levels of query activity where the server might be
> processing 500-1500 queries/second or more spread out over multiple
> backends.
>
> I'll look into this issue more over next several days.  I'll dip back
> into the code and see if I can come up with a better
> answer...unfortunately I can't run the stats collector on until I can
> schedule another load test.

OK, the big problem is that we are nearing RC1.  We would like some
feedback on this as soon as possible.  A major Win32 cleanup for this
could delay the 8.0 release.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Merlin Moncure wrote:
>> Ok, I am starting to strongly suspect the statistics collector of
>> various kinds of malfeasance.

> OK, the big problem is that we are nearing RC1.  We would like some
> feedback on this as soon as possible.  A major Win32 cleanup for this
> could delay the 8.0 release.

I would say that it shouldn't delay the release --- worst case, we say
"the collector doesn't work very well under Win32 yet".  It's probably
not the only part of the system we'll find needs work under Win32.

This is moot if Merlin can find some simple fixable bug, but I'm worried
that doing anything significant might require major work.

BTW, what about the issue we just identified with piperead() failing to
set errno on Windows?  That would certainly account for the "random
collector restarts" complaint ...

            regards, tom lane