>Of course the flaw in this reasoning is that it assumes the OS isn't
>getting in the way. On the HPUX system I've been testing on, the
>performance does seem to be about the same, but evidently it's much
>worse on your system. (Exactly what OS are you running, anyway, and
>on what hardware?) I speculate that your OS is applying some sort of
>read-ahead algorithm that is getting hopelessly confused by lots of
>seeks within a single file. Perhaps it's reading the next block in
>sequence after every program-requested read, and then throwing away that
>work when it sees the program lseek the file instead of reading.
Ok. Here are my settings.
RedHat Linux 6.0 (kernel 2.2.5-smp)
Pentium III 500MHz x 2
RAM: 512MB
Disk: Ultra Wide SCSI 9GB x 4 + Hardware RAID (RAID 5).
Also, I could provide testing scripts to reproduce my tests.
>Next question is what to do about it. I don't suppose we have any way
>of turning off the OS' read-ahead algorithm :-(. We could forget about
>this space-recycling improvement and go back to separate temp files.
>The objection to that, of course, is that while sorting might be faster,
>it doesn't matter how fast the algorithm is if you don't have the disk
>space to execute it.
>
>A possible compromise is to use separate temp files but drop the
>polyphase merge and go to a balanced merge, which'd still access each
>temp file sequentially but would have only a 2X space penalty instead of
>4X (since all the data starts on one set of tapes and gets copied to the
>other set during a complete merge pass). The balanced merge is a little
>slower than polyphase --- more merge passes --- but the space savings
>probably justify it.
I think it depends on the disk space available. Ideally it should be
able to choice the sort algorithm. If it's impossible, the algorithm
that requires least sort space requires would be the way we go. Since
the performance problem only occurs when a table is huge.
>One thing I'd like to know before we make any decisions is whether
>this problem is widespread. Can anyone else run performance tests
>of the speed of large sorts, using current sources vs. 6.5.* ?
I will test with 6.5.2 again.
--
Tatsuo Ishii
