Thread: FreeBSD problem under heavy load
I seem to run into a serious problem. With 6.5.x + FreeBSD 3.2, I get a core under heavy load (16 or more concurrent users). FreeBSD 2.2.x seems more stable but soon or later same thing happens. Examing a core, I found it segfaulted in hash_search(). It was not possible to get more info having a -g compiled backend becasue it did not fail if - g was given. It is likely that random memory corruptions occured. It is also reported by a user that he often sees: NOTICE: LockReplace: xid table corrupted Note that these problems never happen on Linux (even running 128 users are ok on Linux). Only FreeBSD is suffered as far as I can see(not sure about other *BSD). Increasing shmem or semaphore never helps. How to reproduce the problem: 1) obtain pgbench source from ftp.sra.co.jp/pub/cmd/postgres/pgbench/pgbench-1.1.tar.gz 2) unpack the archive and run configure 3) edit the first line in Makefile POSTGRESHOME = /usr/local/pgsql 4) make. you will get an executable "pgbench" there. 5) make a fresh DB (suppose it is "test") 6) initialize DB pgbench -i test this will take for a while 7) run the test pgbench -n -c numeber_of_concurrent users test I see problems with numeber_of_concurrent users ~ 16 or more. Here are my shmem settings: shminfo: shmmax: 41943041 (max shared memory segment size) shmmin: 1 (min shared memory segmentsize) shmmni: 32 (max number of shared memory identifiers) shmseg: 8 (max shared memory segmentsper process) shmall: 10240 (max amount of shared memory in pages) seminfo: semmap: 40 (# of entries in semaphore map) semmni: 32 (# of semaphore identifiers) semmns: 256 (# of semaphores in system) semmnu: 30 (# of undo structures in system) semmsl: 256 (max# of semaphores per id) semopm: 100 (max # of operations per semop call) semume: 10 (max # of undoentries per process) semusz: 92 (size in bytes of undo structure) semvmx: 32767 (semaphore maximumvalue) semaem: 16384 (adjust on exit max value) Any thoughts? -- Tatsuo Ishii
Note that same phenomen happens on current too. -- Tatsuo Ishii > I seem to run into a serious problem. With 6.5.x + FreeBSD 3.2, I get > a core under heavy load (16 or more concurrent users). FreeBSD 2.2.x > seems more stable but soon or later same thing happens. Examing a > core, I found it segfaulted in hash_search(). It was not possible to > get more info having a -g compiled backend becasue it did not fail if - > g was given. It is likely that random memory corruptions occured. It > is also reported by a user that he often sees: > > NOTICE: LockReplace: xid table corrupted > > Note that these problems never happen on Linux (even running 128 users > are ok on Linux). Only FreeBSD is suffered as far as I can see(not > sure about other *BSD). Increasing shmem or semaphore never helps. > > How to reproduce the problem: > > 1) obtain pgbench source from > ftp.sra.co.jp/pub/cmd/postgres/pgbench/pgbench-1.1.tar.gz > > 2) unpack the archive and run configure > > 3) edit the first line in Makefile > > POSTGRESHOME = /usr/local/pgsql > > 4) make. you will get an executable "pgbench" there. > > 5) make a fresh DB (suppose it is "test") > > 6) initialize DB > > pgbench -i test > > this will take for a while > > 7) run the test > > pgbench -n -c numeber_of_concurrent users test > > I see problems with numeber_of_concurrent users ~ 16 or more. > > Here are my shmem settings: > > shminfo: > shmmax: 41943041 (max shared memory segment size) > shmmin: 1 (min shared memory segment size) > shmmni: 32 (max number of shared memory identifiers) > shmseg: 8 (max shared memory segments per process) > shmall: 10240 (max amount of shared memory in pages) > > seminfo: > semmap: 40 (# of entries in semaphore map) > semmni: 32 (# of semaphore identifiers) > semmns: 256 (# of semaphores in system) > semmnu: 30 (# of undo structures in system) > semmsl: 256 (max # of semaphores per id) > semopm: 100 (max # of operations per semop call) > semume: 10 (max # of undo entries per process) > semusz: 92 (size in bytes of undo structure) > semvmx: 32767 (semaphore maximum value) > semaem: 16384 (adjust on exit max value) > > Any thoughts? > -- > Tatsuo Ishii > > ************
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> I seem to run into a serious problem. With 6.5.x + FreeBSD 3.2, I get
> a core under heavy load (16 or more concurrent users). FreeBSD 2.2.x
> seems more stable but soon or later same thing happens. Examing a
> core, I found it segfaulted in hash_search().
I've been looking into this without much success. I cannot reproduce it
here under HPUX --- I ran pgbench for several hours without seeing any
problem. I also made another pass over the dynahash.c code looking for
portability bugs, but didn't find anything that looked promising. (The
code is ugly and fragile, but AFAICT it will work under existing usage
patterns.) It's quite possible the problem is elsewhere and dynahash is
just on the receiving end of a memory clobber ... but if so, we have
very little to go on in guessing where to look.
Can anyone else reproduce the problem? Does anything show up in the
postmaster log at or just before the crash?
regards, tom lane
PS: pgbench's configure fails on HPUX, because HP's compiler doesn't
like whitespace before #include. I modified configure.in like this:
AC_TRY_LINK([#include <sys/time.h>
#include <sys/resource.h>], [struct rlimit rlim;
> Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> > I seem to run into a serious problem. With 6.5.x + FreeBSD 3.2, I get
> > a core under heavy load (16 or more concurrent users). FreeBSD 2.2.x
> > seems more stable but soon or later same thing happens. Examing a
> > core, I found it segfaulted in hash_search().
>
> I've been looking into this without much success. I cannot reproduce it
> here under HPUX --- I ran pgbench for several hours without seeing any
> problem. I also made another pass over the dynahash.c code looking for
> portability bugs, but didn't find anything that looked promising. (The
> code is ugly and fragile, but AFAICT it will work under existing usage
> patterns.) It's quite possible the problem is elsewhere and dynahash is
> just on the receiving end of a memory clobber ... but if so, we have
> very little to go on in guessing where to look.
>
> Can anyone else reproduce the problem? Does anything show up in the
> postmaster log at or just before the crash?
>
> regards, tom lane
I think I got it. in storage/lmgr/lock.c:WaitOnLock:
char old_status[64], new_status[64]; : :
strcpy(old_status, PS_STATUS);strcpy(new_status, PS_STATUS);strcat(new_status, " waiting");PS_SET_STATUS(new_status);
: :PS_SET_STATUS(old_status);
The current status string is copied into old_status, then the pointer
to it is set to a gloable variable ps_status by PS_SET_STATUS
macro. Unfortunately old_status is on the stack, so once WaitOnLock
returns, ps_status would point to a garbage. In the subsequent call to
WaitOnLock,
strcpy(old_status, PS_STATUS);
will copy garbage string into old_status. So if the string is longer
than 63, the stack would be broken. Note that this would not happen on
Linux due to the difference of the definition of the macro. See
include/utils/ps_status.h for more details.
Also, I don't understand why:
strcpy(new_status, PS_STATUS);strcat(new_status, " waiting");PS_SET_STATUS(new_status);
is necessary. Just:
PS_SET_STATUS("waiting");
should be enough. After doing some tests on my FreeBSD and Linux box,
I will commit fixes to both current and 6.5 source tree.
> PS: pgbench's configure fails on HPUX, because HP's compiler doesn't
> like whitespace before #include. I modified configure.in like this:
>
> AC_TRY_LINK([#include <sys/time.h>
> #include <sys/resource.h>],
> [struct rlimit rlim;
Thanks. I will incorporate your fix.
BTW, I think pgbench is usefull to detect this kind of problems. Can I
put it into contrib or somewhere?
--
Tatsuo Ishii
Tatsuo Ishii <t-ishii@sra.co.jp> writes:
> The current status string is copied into old_status, then the pointer
> to it is set to a gloable variable ps_status by PS_SET_STATUS
> macro. Unfortunately old_status is on the stack, so once WaitOnLock
> returns, ps_status would point to a garbage. In the subsequent call to
> WaitOnLock,
> strcpy(old_status, PS_STATUS);
> will copy garbage string into old_status. So if the string is longer
> than 63, the stack would be broken. Note that this would not happen on
> Linux due to the difference of the definition of the macro. See
> include/utils/ps_status.h for more details.
Ugh. It wouldn't happen on HPUX either, because the PS_STATUS stuff
all compiles as no-ops here. So that's why I couldn't see it.
You didn't say what you had in mind to fix this, but I think the safest
approach would be to reserve an area to copy the PS_SET_STATUS string
into on *all* systems. Otherwise we'll just get bitten by this kind of
bug again in future.
> BTW, I think pgbench is usefull to detect this kind of problems. Can I
> put it into contrib or somewhere?
Sounds like a good idea to me.
regards, tom lane
> You didn't say what you had in mind to fix this, but I think the safest > approach would be to reserve an area to copy the PS_SET_STATUS string > into on *all* systems. Otherwise we'll just get bitten by this kind of > bug again in future. Done for current. > > BTW, I think pgbench is usefull to detect this kind of problems. Can I > > put it into contrib or somewhere? > > Sounds like a good idea to me. Will commit into contrib. -- Tatsuo Ishii
On 1999-12-10, Tatsuo Ishii mentioned: > BTW, I think pgbench is usefull to detect this kind of problems. Can I > put it into contrib or somewhere? Under src/test there already is a bench subdirectory which I'm not sure what it is for, but pgbench might have good company there. -- Peter Eisentraut Sernanders väg 10:115 peter_e@gmx.net 75262 Uppsala http://yi.org/peter-e/ Sweden