Thread: Solaris

Solaris

From
"Ian Harding"
Date:
We have an Ultra Enterprise 3000 with a Sparc Array 1010 lying around here acting as a coffee table in the server room.
It was being used as a database server, but when a drive crumped, nobody knew how to get the array put back together,
sosome $250 per hour consultant wiped out all the data.  The solution was, of course, to switch to MS SQL Server on
i386stuff and use the Sun machine as a coffee table. 

It seem like a bit of a waste.

I am running my PostgreSQL database on a Dell PowerEdge 2400.  What, advantages/disadvantages are there to learning
Solarisand migrating my stuff over to the coffee table machine?  Does the Dell fall under the heading of "cheap"
(crummy)hardware that Tom alluded to causing corruption issues? 


Ian Harding
Programmer/Analyst II
Tacoma-Pierce County Health Department
iharding@tpchd.org
(253) 798-3549


Re: Solaris

From
Shridhar Daithankar
Date:
On Wednesday 23 April 2003 20:33, Ian Harding wrote:
> I am running my PostgreSQL database on a Dell PowerEdge 2400.  What,
> advantages/disadvantages are there to learning Solaris and migrating my
> stuff over to the coffee table machine?  Does the Dell fall under the
> heading of "cheap" (crummy) hardware that Tom alluded to causing corruption
> issues?

Dell server might be good enough for most tasks but if the sun machine is 64
bit, then you have a significant advantage there. Besides they should be bit
more robust(Sorry, this is absolute wild shot. Have no idea about specs of
any of these two machines.)

Just one real advice, from what I have heard on lists. If the sun machine
supports, install linux rather than solaris. Apparently linux on sparc is far
faster than solaris, at least at lower to middle end. Besides postgresql and
solaris have had performance issues in past.

 Shridhar


Re: Solaris

From
"scott.marlowe"
Date:
On Wed, 23 Apr 2003, Shridhar Daithankar wrote:

> On Wednesday 23 April 2003 20:33, Ian Harding wrote:
> > I am running my PostgreSQL database on a Dell PowerEdge 2400.  What,
> > advantages/disadvantages are there to learning Solaris and migrating my
> > stuff over to the coffee table machine?  Does the Dell fall under the
> > heading of "cheap" (crummy) hardware that Tom alluded to causing corruption
> > issues?
>
> Dell server might be good enough for most tasks but if the sun machine is 64
> bit, then you have a significant advantage there. Besides they should be bit
> more robust(Sorry, this is absolute wild shot. Have no idea about specs of
> any of these two machines.)
>
> Just one real advice, from what I have heard on lists. If the sun machine
> supports, install linux rather than solaris. Apparently linux on sparc is far
> faster than solaris, at least at lower to middle end. Besides postgresql and
> solaris have had performance issues in past.

Even better, throw in an extra drive and configure it to dual boot.  Then
you can test the two against each other be sure which is faster for what
you're doing.


Re: Solaris

From
Richard Welty
Date:
On Wed, 23 Apr 2003 20:44:02 +0530 Shridhar Daithankar <shridhar_daithankar@persistent.co.in> wrote:
> Just one real advice, from what I have heard on lists. If the sun
> machine
> supports, install linux rather than solaris. Apparently linux on sparc
> is far
> faster than solaris, at least at lower to middle end. Besides postgresql
> and
> solaris have had performance issues in past.

investigate OpenBSD and NetBSD as well (although if the Sun has multiple
processors, which is likely with an E, OpenBSD is no longer a candidate due
to lack of SMP support.) the *BSD systems are generally very stable and
reliable, and have a couple of advantages over Linux in a number of
situations.

richard
--
Richard Welty                                         rwelty@averillpark.net
Averill Park Networking                                         518-573-7592
              Unix, Linux, IP Network Engineering, Security

Re: Solaris

From
Hadley Willan
Date:
You should at least have ECC RAM in the Solaris box, but then again, you
should have ECC RAM in any "server" to avoid in memory corruption.

If the disk "Array" blew it's toys, then it sounds like you may have
yourself a RAID controller, you could easily buy a few new hard disks,
and get yourself back up and running. RAID 5 needs a minimum of three
disks, RAID 1 Needs two. Failing the RAID controller, the solaris will
have a SCSI subsystem .... mmmmm, 15,000rpm, 16MB Cache disks....

Hadley.

On Thu, 2003-04-24 at 03:34, scott.marlowe wrote:
> On Wed, 23 Apr 2003, Shridhar Daithankar wrote:
>
> > On Wednesday 23 April 2003 20:33, Ian Harding wrote:
> > > I am running my PostgreSQL database on a Dell PowerEdge 2400.  What,
> > > advantages/disadvantages are there to learning Solaris and migrating my
> > > stuff over to the coffee table machine?  Does the Dell fall under the
> > > heading of "cheap" (crummy) hardware that Tom alluded to causing corruption
> > > issues?
> >
> > Dell server might be good enough for most tasks but if the sun machine is 64
> > bit, then you have a significant advantage there. Besides they should be bit
> > more robust(Sorry, this is absolute wild shot. Have no idea about specs of
> > any of these two machines.)
> >
> > Just one real advice, from what I have heard on lists. If the sun machine
> > supports, install linux rather than solaris. Apparently linux on sparc is far
> > faster than solaris, at least at lower to middle end. Besides postgresql and
> > solaris have had performance issues in past.
>
> Even better, throw in an extra drive and configure it to dual boot.  Then
> you can test the two against each other be sure which is faster for what
> you're doing.
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
--
Hadley Willan > Systems Development > Deeper Design Limited. +64(7)377-3328
hadley.willan@deeperdesign.co.nz > www.deeperdesign.com > +64(21)-28-41-463
Level 1, 4 Tamamutu St, PO Box 90, TAUPO 2730, New Zealand.


Re: Solaris

From
Mark Kirkwood
Date:
Shridhar Daithankar wrote:

>Just one real advice, from what I have heard on lists. If the sun machine
>supports, install linux rather than solaris. Apparently linux on sparc is far
>faster than solaris, at least at lower to middle end.
>
Any idea how much faster (and in what areas)?

>Besides postgresql and
>solaris have had performance issues in past.
>
>
I believe this is all *sorted* now :-) The issue was the qsort library
bundled with Solaris - Postgresql now supplies its own. So Solaris is ok
for Postgresql (I think quite a few folk actually use this combination).

best wishes

Mark


Re: Solaris

From
Andrew Sullivan
Date:
On Fri, Apr 25, 2003 at 02:03:24PM +1200, Mark Kirkwood wrote:
> I believe this is all *sorted* now :-) The issue was the qsort library
> bundled with Solaris - Postgresql now supplies its own. So Solaris is ok
> for Postgresql (I think quite a few folk actually use this combination).

It's still not all that fast.  Better, but not what you'd expect.
There's something fishy with the SYSV shared memory management, but
I'm darned if I can figure out what it is.

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: Solaris

From
Tom Lane
Date:
Andrew Sullivan <andrew@libertyrms.info> writes:
> There's something fishy with the SYSV shared memory management, but
> I'm darned if I can figure out what it is.

Hmm, you're still running 7.2.*, right?  There's some code added in 7.3
to enable "intimate shared memory" on Solaris:

#if defined(solaris) && defined(__sparc__)
    /* use intimate shared memory on SPARC Solaris */
    memAddress = shmat(shmid, 0, SHM_SHARE_MMU);
#else
    memAddress = shmat(shmid, 0, 0);
#endif

I disremember the details but we were told this would improve
performance.  It'd be an easy enough patch in 7.2
(src/backend/storage/ipc/ipc.c about line 638) if you care to try it.

            regards, tom lane


Re: Solaris

From
Andrew Sullivan
Date:
On Thu, Apr 24, 2003 at 11:08:11PM -0400, Tom Lane wrote:
> Hmm, you're still running 7.2.*, right?  There's some code added in 7.3
> to enable "intimate shared memory" on Solaris:

Tried it.  I was unable to show that it helped, and it seemed to make
things worse in some cases.  As usual, the time I actually got to
spend doing this was (it felt) inversely proportionate to its value,
which means that any numbers I managed to produce are not real
reliable.  But it sure didn't have the effect I was hoping for.  The
best I can say is that the numbers were inconclusive for our
application.  I have some more tests planned, if I can find the time.

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: Solaris

From
Mark Kirkwood
Date:
Andrew Sullivan wrote:

>It's still not all that fast.  Better, but not what you'd expect.
>There's something fishy with the SYSV shared memory management, but
>I'm darned if I can figure out what it is.
>
>
Just out of curiousity,  where are you seeing the effect of this the most?
(Running through my mind is the thought that VM management in Linux is
not so hot either....)

cheers
Mark


Re: Solaris

From
"scott.marlowe"
Date:
On Fri, 25 Apr 2003, Mark Kirkwood wrote:

> Shridhar Daithankar wrote:
>
> >Just one real advice, from what I have heard on lists. If the sun machine
> >supports, install linux rather than solaris. Apparently linux on sparc is far
> >faster than solaris, at least at lower to middle end.
> >
> Any idea how much faster (and in what areas)?

My general experience from a year ago was that Linux was about 2 to 3
times faster on older 32 bit sparc hardware than Solaris, and just under 2
times as fast on 64 bit hardware.  This is with one CPU.  I'd expect a 64
CPU E10k to run postgresql faster running Solaris rather than Linux, but
until I can come up with a spare $24k to buy a used one on Ebay I won't
know. :-)

> >Besides postgresql and
> >solaris have had performance issues in past.
> >
> >
> I believe this is all *sorted* now :-) The issue was the qsort library
> bundled with Solaris - Postgresql now supplies its own. So Solaris is ok
> for Postgresql (I think quite a few folk actually use this combination).

Mostly.  The issue in the past was mainly that Solaris had a brain damaged
sort() call that was very slow when it had a lot of duplicate keys.  That
has since been updated by Sun.

However, Solaris' heavy process / light thread design is probably sub
optimal for Postgresql in an environment where you are forking the server
all the time.  If you have pooled connections, then Solaris should do
fine.  I'd love to get ahold of an older SMP Sparc box and test the two
against each other.


Re: Solaris

From
Mark Kirkwood
Date:
scott.marlowe wrote:

>My general experience from a year ago was that Linux was about 2 to 3
>times faster on older 32 bit sparc hardware than Solaris, and just under 2
>times as fast on 64 bit hardware.
>
Wow - 2 times faster is significant ! It would be interesting to try a
midrange SMP box ( e.g E280/E480 ).

>  The issue in the past was mainly that Solaris had a brain damaged
>sort() call that was very slow when it had a lot of duplicate keys.  That
>has since been updated by Sun.
>
>
Hmm - I ran into that situation with qsort() and many equal keys (see
Hackers thread "Solaris Performance"). Sadly Sun have not amended that
situation at all (in Solaris 8 anyway).

best wishes

Mark


Re: Solaris

From
Andrew Sullivan
Date:
On Fri, Apr 25, 2003 at 11:19:47PM +1200, Mark Kirkwood wrote:
> Just out of curiousity,  where are you seeing the effect of this the most?
> (Running through my mind is the thought that VM management in Linux is
> not so hot either....)

The problems seem to be somewhat related to load.  With no load, of
course, everything is real fast.  But with any sort of load, we get
occasional spikes of unexpectedly slow performance, even on things
like INSERT.  Some of this I have been able to attribute to
maintenance issues, but not all of it.  We see from truss, however,
that a significant amount of timee is spent in the SYSV system
calls, so it appears that there is some kind of inefficiency there.
Sorry I'm not able to provide more detail; as I say, I'm not real
happy with the extent of the tests I've been able to conduct, and in
any case, the problems aren't reproducable at will.

A

--
----
Andrew Sullivan                         204-4141 Yonge Street
Liberty RMS                           Toronto, Ontario Canada
<andrew@libertyrms.info>                              M2P 2A8
                                         +1 416 646 3304 x110


Re: Solaris

From
"scott.marlowe"
Date:
On Sat, 26 Apr 2003, Mark Kirkwood wrote:

> scott.marlowe wrote:
>
> >My general experience from a year ago was that Linux was about 2 to 3
> >times faster on older 32 bit sparc hardware than Solaris, and just under 2
> >times as fast on 64 bit hardware.
> >
> Wow - 2 times faster is significant ! It would be interesting to try a
> midrange SMP box ( e.g E280/E480 ).

Keep in mind, this is running Postgresql that measures out at about twice
as fast.  I was testing with things like the 10k row database in the
regression tests, a few of our own 1M row tables, and pgbench with
parallel access up to a good amount (I think about 64 or so)  The boxes
were an Ultra 1, and a Sparc 20.  The Sparc 20 / Linux was running Solaris
and Linux in dual boot, and the linux performance on it was astounding,
Solaris was painfully slow at everything.

On the Ultra 1, the race was much closer, i.e. Linux was about 80% faster
than solaris.  I think we were running whatever the latest production
quality versions of RedHat (6.2, the last Sparc version) and solaris at
the time.

note that we're talking about a year and a half or so ago, so things may
have changed under both OSes.

> >  The issue in the past was mainly that Solaris had a brain damaged
> >sort() call that was very slow when it had a lot of duplicate keys.  That
> >has since been updated by Sun.
> >
> >
> Hmm - I ran into that situation with qsort() and many equal keys (see
> Hackers thread "Solaris Performance"). Sadly Sun have not amended that
> situation at all (in Solaris 8 anyway).

Actually, I thought there was patch out there somewhere.  Anyone else
know?


Re: Solaris

From
Bruce Momjian
Date:
scott.marlowe wrote:
> note that we're talking about a year and a half or so ago, so things may
> have changed under both OSes.
>
> > >  The issue in the past was mainly that Solaris had a brain damaged
> > >sort() call that was very slow when it had a lot of duplicate keys.  That
> > >has since been updated by Sun.
> > >
> > >
> > Hmm - I ran into that situation with qsort() and many equal keys (see
> > Hackers thread "Solaris Performance"). Sadly Sun have not amended that
> > situation at all (in Solaris 8 anyway).
>
> Actually, I thought there was patch out there somewhere.  Anyone else
> know?

qsort() patch is in PostgreSQL 7.3.  We used the FreeBSD one for Solaris.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Re: Solaris

From
Mark Kirkwood
Date:
scott.marlowe wrote:

>Actually, I thought there was patch out there somewhere.  Anyone else
>know?
>
>
>
It looks like Solaris 9 has this included in a libc patch:
http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fpatches/112874

However I think the solution chosen by the Postgresql developers  -
using qsort.c from BSD is the best one, as it *sorts* all the Solaris
releases (its in your Pg source : src/port/qsort.c).

In fact I wonder if using the BSD qsort for *all* platforms might be a
good idea - it would provide cross platform consistency and possibly
better performance (e.g. It was several times quicker than  glibc qsort,
when I checked this on Linux last year.... )

best wishes
Mark


Re: Solaris

From
Mark Kirkwood
Date:
Andrew Sullivan wrote:

>We see from truss, however,
>that a significant amount of timee is spent in the SYSV system
>calls, so it appears that there is some kind of inefficiency there.
>
It would be interesting to run a profiling enabled postgres binary -
altho that's not really nice on a production system :-).

>
>in any case, the problems aren't reproducable at will.
>
>
The best ones never seem to be - means you can't try the profiling in a
safe environment....:-(

regards

Mark


Re: Solaris

From
Bruce Momjian
Date:
Mark Kirkwood wrote:
> scott.marlowe wrote:
>
> >Actually, I thought there was patch out there somewhere.  Anyone else
> >know?
> >
> >
> >
> It looks like Solaris 9 has this included in a libc patch:
> http://sunsolve.sun.com/pub-cgi/retrieve.pl?doc=fpatches/112874
>
> However I think the solution chosen by the Postgresql developers  -
> using qsort.c from BSD is the best one, as it *sorts* all the Solaris
> releases (its in your Pg source : src/port/qsort.c).
>
> In fact I wonder if using the BSD qsort for *all* platforms might be a
> good idea - it would provide cross platform consistency and possibly
> better performance (e.g. It was several times quicker than  glibc qsort,
> when I checked this on Linux last year.... )

Replacing qsort() on all platforms has a "we know better than the OS"
feel to it that we try to avoid.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


qsort (was Re: Solaris)

From
Tom Lane
Date:
Bruce Momjian <pgman@candle.pha.pa.us> writes:
> Mark Kirkwood wrote:
>> In fact I wonder if using the BSD qsort for *all* platforms might be a
>> good idea - it would provide cross platform consistency and possibly
>> better performance (e.g. It was several times quicker than  glibc qsort,
>> when I checked this on Linux last year.... )

> Replacing qsort() on all platforms has a "we know better than the OS"
> feel to it that we try to avoid.

I agree on that --- but when it's provable that we do know better than a
*particular* OS, dropping in the BSD qsort seems like an easy win.  Can
anyone back up Mark's finding that the BSD qsort is quicker than glibc's?

            regards, tom lane


Re: qsort (was Re: Solaris)

From
dalgoda@ix.netcom.com (Mike Castle)
Date:
In article <26779.1051629468@sss.pgh.pa.us>,
Tom Lane  <tgl@sss.pgh.pa.us> wrote:
>I agree on that --- but when it's provable that we do know better than a
>*particular* OS, dropping in the BSD qsort seems like an easy win.  Can
>anyone back up Mark's finding that the BSD qsort is quicker than glibc's?

Better yet:  Anyway of running performance tests from configure?

Would a simple counter in the compare function be sufficient to determine
the speed?

mrc

--
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc


Re: qsort (was Re: Solaris)

From
Tom Lane
Date:
dalgoda@ix.netcom.com (Mike Castle) writes:
> Better yet:  Anyway of running performance tests from configure?

Peter will object to that because of cross-compilation issues; and I'll
object because I run configure often enough that I don't want it to take
the time that would be needed for a reliable performance test ...

            regards, tom lane


Re: qsort (was Re: Solaris)

From
dalgoda@ix.netcom.com (Mike Castle)
Date:
In article <4896.1051644271@sss.pgh.pa.us>,
Tom Lane  <tgl@sss.pgh.pa.us> wrote:
>dalgoda@ix.netcom.com (Mike Castle) writes:
>> Better yet:  Anyway of running performance tests from configure?
>
>Peter will object to that because of cross-compilation issues; and I'll
>object because I run configure often enough that I don't want it to take
>the time that would be needed for a reliable performance test ...


Understandable.

What about --with-pg-qsort (that defaults to use for currently known
systems) with a test program people could run if they want?

In that case, would counting the calls to the compare function be the
appropriate measurement (I'd think either wall or system time would vary
too widely).

mrc
--
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc


Re: qsort (was Re: Solaris)

From
Bruce Momjian
Date:
Mike Castle wrote:
> In article <4896.1051644271@sss.pgh.pa.us>,
> Tom Lane  <tgl@sss.pgh.pa.us> wrote:
> >dalgoda@ix.netcom.com (Mike Castle) writes:
> >> Better yet:  Anyway of running performance tests from configure?
> >
> >Peter will object to that because of cross-compilation issues; and I'll
> >object because I run configure often enough that I don't want it to take
> >the time that would be needed for a reliable performance test ...
>
>
> Understandable.
>
> What about --with-pg-qsort (that defaults to use for currently known
> systems) with a test program people could run if they want?

Let's have folks run a test program and get the results for some OS's.
I would prefer to get some results before moving to a formalized option.

> In that case, would counting the calls to the compare function be the
> appropriate measurement (I'd think either wall or system time would vary
> too widely).

No.  It is not calls the compare function, but total time in the qsort
routine that has to be measured.

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073


Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
Bruce Momjian wrote:

>Let's have folks run a test program and get the results for some OS's.
>I would prefer to get some results before moving to a formalized option.
>
>
>
I retested Linux 2.4.19 (Mandrake 9) using a simple "many equal keys"
program (see below) compiled with and without BSD qsort.c from Pg source.

$ time ./testbsdqsort
0.80user 0.02system 0:00.83elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (69major+991minor)pagefaults 0swaps
$ time ./testqsort
2.08user 0.04system 0:02.13elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (105major+1967minor)pagefaults 0swaps


So for the many equal keys, BSD qsort is significantly faster.

Clearly there are other test scenarios to examine, but the many equal
keys case is important.

cheers

Mark


-- the program...

#include <stdio.h>
#include <stdlib.h>

const int elements = 1000000;
const int mod = 100;

int sortfunc( const void* n1, const void* n2 ) {
        int ret;
        ret =  (*(int*)n1) -(*(int*)n2) ;
        return ret;
}

int main(int argc, char** argv ) {
        int array[elements];
        int i;
        for ( i = 0; i < elements; i++ ) {
                array[i] = i % mod;
        }
        qsort( array, elements, sizeof(int),sortfunc);
        return 0;
}


-- end


Re: qsort (was Re: Solaris)

From
Martijn van Oosterhout
Date:
On Wed, Apr 30, 2003 at 01:26:05PM +1200, Mark Kirkwood wrote:
> So for the many equal keys, BSD qsort is significantly faster.
>
> Clearly there are other test scenarios to examine, but the many equal
> keys case is important.

I just whipped up a really simply test package (attached, 3KB). Just unpack
it and run make. Anyway, on a Debian Woody machine here I get:

10^6 items, mod = 100

Value (i = index)       BSD      GLIBC (seconds)
random() * mod          1.58     2.29
i % mod                 0.52     1.69
i / (ITEMS / mod)       0.38     1.09
i ^ 0x5555555           1.18     1.67

It would seem to support the theory that the BSD qsort is faster than the
Glibc one. The results seem quite repeatable (repeats give +/- 5%).
--
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> "the West won the world not by the superiority of its ideas or values or
> religion but rather by its superiority in applying organized violence.
> Westerners often forget this fact, non-Westerners never do."
>   - Samuel P. Huntington

Attachment

Re: qsort (was Re: Solaris)

From
Tom Lane
Date:
dalgoda@ix.netcom.com (Mike Castle) writes:
> What about --with-pg-qsort (that defaults to use for currently known
> systems) with a test program people could run if they want?

I'd lean to "--with-pg-qsort" forcing the BSD qsort, "--without-pg-qsort"
forcing the native qsort, and if you don't say either then you get a
choice based on which OS you are running.  (Not sure how hard this is
to do in the autoconf structure, but if we can do it, it seems like the
right user interface.)

> In that case, would counting the calls to the compare function be the
> appropriate measurement (I'd think either wall or system time would vary
> too widely).

I'd trust overall time measurements way more than call counts.
Obviously it's up to the user to hold overall system load and suchlike
outside factors constant while conducting the tests --- but measuring
any single component like comparison-function calls is just asking to be
misled.

            regards, tom lane


Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
Martijn van Oosterhout wrote:

> Anyway, on a Debian Woody machine here I get:
>
>10^6 items, mod = 100
>
>Value (i = index)       BSD      GLIBC (seconds)
>random() * mod          1.58     2.29
>i % mod                 0.52     1.69
>i / (ITEMS / mod)       0.38     1.09
>i ^ 0x5555555           1.18     1.67
>
>It would seem to support the theory that the BSD qsort is faster than the
>Glibc one. The results seem quite repeatable (repeats give +/- 5%).
>
Martin's test on two Linux machines here gives :

10^6 items, mod = 100

1) Mandrake 9 (x86 - Linux 2.4.19, glibc 2.2.5, gcc 3.2)

Value (i = index)       BSD      GLIBC (seconds)
random() * mod          2.20     2.70
i % mod                 0.80     2.10
i / (ITEMS / mod)       0.53     1.28
i ^ 0x5555555           1.64     1.95

2) Yellowdog 2.3 (ppc - Linux 2.4.19, glibc 2.2.5. gcc 2.95.4)

Value (i = index)       BSD      GLIBC (seconds)
random() * mod          1.74     3.12
i % mod                 0.69     2.65
i / (ITEMS / mod)       0.55     1.66
i ^ 0x5555555           1.42     2.45

Interestingly, the the ppc machine gets more improvement from the BSD
qsort than the x86 one does.

regards

Mark


Re: qsort (was Re: Solaris)

From
Jean-Christian Imbeault
Date:
  More test values :)

  On a RH 8.0 i686 box with a custom 2.4.20 kernel, with gcc 3.2 20020903
  (Red Hat Linux 8.0 3.2-7) and with glibc 2.2.93

  Value (i = index)        BSD    GLIBC
  random() * mod      0.53    0.78
  i % mod            0.15    0.64
  i / (ITEMS / mod)    0.10    0.35
  i ^ 0x5555555        0.39    0.54




--

Jean-Christian Imbeault


Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
I prevailed on some friends to try out a non Redhat distro plus HPUX 10.20 :

1 United Linux 1.0 (x86 Linux 2.4.19 )

Value (i = index)       BSD      GLIBC (seconds)
random() * mod          0.35     0.56
i % mod                 0.14     0.41
i / (ITEMS / mod)       0.10     0.27
i ^ 0x5555555           0.26     0.42

2 HPUX 10.20 (pa-risc1.1 D380 )

Value (i = index)       BSD      HPUX (seconds)
random() * mod          8.3      14.0
i % mod                 2.9      4.9
i / (ITEMS / mod)       2.4      2.9
i ^ 0x5555555           7.8      13.8

The HPUX results are interesting !

regards

Mark


Re: qsort (was Re: Solaris)

From
Dennis Gearon
Date:
The trend has been approx:

    On Linux
        BSD qsort is 30-55% faster on average.
    On Non Liux
        BSD qsor it 50-90% on average.

This kind of says that Linux is doing something right in the kernal,
outdoing mainframe software in this sort of thing.

Mark Kirkwood wrote:
>
> I prevailed on some friends to try out a non Redhat distro plus HPUX 10.20 :
>
> 1 United Linux 1.0 (x86 Linux 2.4.19 )
>
> Value (i = index)       BSD      GLIBC (seconds)
> random() * mod          0.35     0.56
> i % mod                 0.14     0.41
> i / (ITEMS / mod)       0.10     0.27
> i ^ 0x5555555           0.26     0.42
>
> 2 HPUX 10.20 (pa-risc1.1 D380 )
>
> Value (i = index)       BSD      HPUX (seconds)
> random() * mod          8.3      14.0
> i % mod                 2.9      4.9
> i / (ITEMS / mod)       2.4      2.9
> i ^ 0x5555555           7.8      13.8
>
> The HPUX results are interesting !
>
> regards
>
> Mark
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)


Re: qsort (was Re: Solaris)

From
Tom Lane
Date:
Mark Kirkwood <markir@paradise.net.nz> writes:
> The HPUX results are interesting !

I can confirm the results for the i % mod case on my HPUX 10.20 system
(I get about 2.65 sec vs 4.40 sec) and also the i ^ 0x5555555 case
(about 7.25 vs 12.85 sec).  Didn't try the other cases...

            regards, tom lane


Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
Dennis Gearon wrote:

>The trend has been approx:
>
>    On Linux
>        BSD qsort is 30-55% faster on average.
>    On Non Liux
>        BSD qsort it 50-90% on average.
>
>This kind of says that Linux is doing something right in the kernal,
>outdoing mainframe software in this sort of thing.
>
The difference may be due to GNU and HP using different qsort algorithm
tweeks. Consider :

+ HPUX vs BSD  - very different for random*mod, close for i / (ITEMS /
mod).
+ Linux vs BSD - reasonably different for all 4 cases.

(Only way to tell would be to eyeball HP's qsort code...I am not sure if
thats possible)

regards

Mark


Re: qsort (was Re: Solaris)

From
dalgoda@ix.netcom.com (Mike Castle)
Date:
In article <19660.1051676560@sss.pgh.pa.us>,
Tom Lane  <tgl@sss.pgh.pa.us> wrote:
>dalgoda@ix.netcom.com (Mike Castle) writes:
>> What about --with-pg-qsort (that defaults to use for currently known
>> systems) with a test program people could run if they want?
>
>I'd lean to "--with-pg-qsort" forcing the BSD qsort, "--without-pg-qsort"
>forcing the native qsort, and if you don't say either then you get a
>choice based on which OS you are running.  (Not sure how hard this is
>to do in the autoconf structure, but if we can do it, it seems like the
>right user interface.)

It's fairly easy, actually.

Something like:

AC_ARG_WITH([pg-qsort],
            AC_HELP_STRING([--with-pg-qsort],
                           [use internal qsort (default is system
dependent)]),
case $withval
        yes) force internal;;
        no) force system;;
        *) tell user to don't do that (--with-pg-qsort=blah);;
esac,
case OS
        *Solaris*) force internal;;
        *) force system;;
esac
)

>> In that case, would counting the calls to the compare function be the
>> appropriate measurement (I'd think either wall or system time would vary
>> too widely).
>
>I'd trust overall time measurements way more than call counts.
>Obviously it's up to the user to hold overall system load and suchlike
>outside factors constant while conducting the tests --- but measuring
>any single component like comparison-function calls is just asking to be
>misled.

I'm not certain.

I ran the tests posted to this lists with a few mods and got some very
interesting results.

First I added a counter to the compare function, and the most cases, the
glibc implementation was called significantly less often than the BSD
compare function.

In a simple test function, like comparing two ints, then yes, the BSD
implementation was faster.  But in a more complex function, say comparing
strings, often times the glibc version was faster.  Why?  Because the
time spent in the compare function became the overwhelming factor.

So, I ask you this:  in places where qsort is used in PG, is it more likely
to be used for simple comparisons or for complex comparisons that involve a
mixture of strings and ints?  And, is it more likely to be called with
datasets that are partially sorted or not?

BSD prevailed on trivial comparisons and on sets that had any type of
sorting to them.  Glibc prevailed on random order with heavier comparisons.

mrc
--
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc


Re: qsort (was Re: Solaris)

From
Tom Lane
Date:
dalgoda@ix.netcom.com (Mike Castle) writes:
> In a simple test function, like comparing two ints, then yes, the BSD
> implementation was faster.  But in a more complex function, say comparing
> strings, often times the glibc version was faster.  Why?  Because the
> time spent in the compare function became the overwhelming factor.

Interesting.

> So, I ask you this:  in places where qsort is used in PG, is it more likely
> to be used for simple comparisons or for complex comparisons that involve a
> mixture of strings and ints?

The only place that I think performance is likely to matter for is the
calls in tuplesort.c, which are expensive --- even if you end up calling
something as cheap as btint4cmp, there's a lot of overhead in between.
(I seem to recall hearing that Oracle expends effort trying to "compile"
code for tuple comparisons, but we don't ... yet ...)

  And, is it more likely to be called with
> datasets that are partially sorted or not?

I think that one's unanswerable.  It'd depend on the workload.

            regards, tom lane


Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
Mike Castle wrote:

>First I added a counter to the compare function, and the most cases, the
>glibc implementation was called significantly less often than the BSD
>compare function.
>
>
Interesting - all examples I tried had glibc compare count higher, do
you want to post one of yours?

>In a simple test function, like comparing two ints, then yes, the BSD
>implementation was faster.  But in a more complex function, say comparing
>strings, often times the glibc version was faster.  Why?  Because the
>time spent in the compare function became the overwhelming factor.
>
>
Considering strings is a good point - I went away and tried some
examples,  unfortunately I only managed to see BSD faster, but the
difference was not as large as in the integer tests.


regards

Mark


Re: qsort (was Re: Solaris)

From
Bruce Momjian
Date:
I assume we decided that BSD sort isn't fast enough to warrant replacing
the native qsort.

---------------------------------------------------------------------------

Mark Kirkwood wrote:
> Mike Castle wrote:
>
> >First I added a counter to the compare function, and the most cases, the
> >glibc implementation was called significantly less often than the BSD
> >compare function.
> >
> >
> Interesting - all examples I tried had glibc compare count higher, do
> you want to post one of yours?
>
> >In a simple test function, like comparing two ints, then yes, the BSD
> >implementation was faster.  But in a more complex function, say comparing
> >strings, often times the glibc version was faster.  Why?  Because the
> >time spent in the compare function became the overwhelming factor.
> >
> >
> Considering strings is a good point - I went away and tried some
> examples,  unfortunately I only managed to see BSD faster, but the
> difference was not as large as in the integer tests.
>
>
> regards
>
> Mark
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

--
  Bruce Momjian                        |  http://candle.pha.pa.us
  pgman@candle.pha.pa.us               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

Re: qsort (was Re: Solaris)

From
Mark Kirkwood
Date:
That is probably the most reasonable conclusion to draw from the results ;-)

cheers

Mark

Bruce Momjian wrote:

>I assume we decided that BSD sort isn't fast enough to warrant replacing
>the native qsort.
>
>---------------------------------------------------------------------------
>
>Mark Kirkwood wrote:
>
>
>>Mike Castle wrote:
>>
>>
>>
>>>First I added a counter to the compare function, and the most cases, the
>>>glibc implementation was called significantly less often than the BSD
>>>compare function.
>>>
>>>
>>>
>>>
>>Interesting - all examples I tried had glibc compare count higher, do
>>you want to post one of yours?
>>
>>
>>
>>>In a simple test function, like comparing two ints, then yes, the BSD
>>>implementation was faster.  But in a more complex function, say comparing
>>>strings, often times the glibc version was faster.  Why?  Because the
>>>time spent in the compare function became the overwhelming factor.
>>>
>>>
>>>
>>>
>>Considering strings is a good point - I went away and tried some
>>examples,  unfortunately I only managed to see BSD faster, but the
>>difference was not as large as in the integer tests.
>>
>>
>>regards
>>
>>Mark
>>
>>
>>---------------------------(end of broadcast)---------------------------
>>TIP 6: Have you searched our list archives?
>>
>>http://archives.postgresql.org
>>
>>
>>
>
>
>