Re: (resolution?) Re: [HACKERS] memory problem again - Mailing list pgsql-hackers

Tom,

I think the #2 is more likely, because:

the kernel is compiled with large enough data size:

# support for larger processes and number of childs
options         "DFLDSIZ=\(128*1024*1024\)"
options         "MAXDSIZ=\(256*1024*1024\)"
options         "CHILD_MAX=256"
options         "OPEN_MAX=256"
options         "KMAPENTRIES=4000" # Prevents kmem malloc errors !
options         "KMEMSIZE=\(32*1024*1024\)"

the default postgres acocunt limits are:

coredumpsize    unlimited
cputime         unlimited
datasize        131072 kbytes
filesize        unlimited
maxproc         256
memorylocked    85380 kbytes
memoryuse       256136 kbytes
openfiles       128
stacksize       2048 kbytes

I run the postmaster after unlimit, which sets limits thus:

coredumpsize    unlimited
cputime         unlimited
datasize        262144 kbytes
filesize        unlimited
maxproc         4116
memorylocked    256140 kbytes
memoryuse       256136 kbytes
openfiles       13196
stacksize       262144 kbytes

I will do some experimentation with the -S flag to see how it works.

BTW, this postgres is compiled with default of 64 backends - I saw recently 
note here that this may interfere with the -S option somehow....

(another not related bug, but still on memory allocation)
Still - this does not explain why postgres cannot allocated more than 76 MB 
(according to top) on BSD/OS (never did, actually - any previous version too), 
while a simple malloc(1 MB) loop allocates up to the process limit.

Maybe at some time postrges tries to allocate 'larger' chunk, which the BSD/OS 
malloc does not like?

Daniel

>>>Tom Lane said:> Daniel Kalchev <daniel@digsys.bg> writes:> > I found out how to resolve this problem, yet it does
notexplain why it > > happens anyway!> > I had postmaster started with this script:> > postmaster
-D/usr/local/pgsql/data-B 256 -i -o "-e -S 8192" >> > > /usr/local/pgsql/errlog 2>&1 &> > Removing all the parameters
topostmaster> > postmaster -D/usr/local/pgsql/data -i -o "-e" >> /usr/local/pgsql/errlog 2    >&1 &> > made it
work....>> Perhaps some memory management problem? I guess the -S option is the culpr    it > > here, but this machine
has256 MB RAM and actually never swaps (yet).> > 8192 * 1K = 8 meg workspace per sort sure doesn't sound unreasonable.>
Thereis a sort going on under-the-hood in your SELECT DISTINCT (it's> implemented in the same fashion as "sort |
uniq"),but under ordinary> circumstances that doesn't cause any problem.  I can see a couple of> possibilities:>     1.
Youhave a very small kernel limit on per-process data space,>        probably 8M or at most 16M.>     2. Something is
brokenin the sort code that makes it fail to>        obey the -S limit.> I favor #1, since if #2 were true we'd
probablyhave noticed it before.> > You might try experimenting with a couple of different -S values (-B> shouldn't make
anydifference here, it just affects the size of the> shared-memory-block request), and watching the size of the
backend>process with top(1) or something like it.> > In the meantime, find out where kernel parameters are set on your>
system,and look at what MAXDSIZ is set to...> >             regards, tom lane