Thread: join problems: FATAL 1: palloc failure: memory exhausted

join problems: FATAL 1: palloc failure: memory exhausted

From
Oleg Bartunov
Date:
Hi,

Below is an example of my query, which returns no plan and produces
FATAL 1:  palloc failure: memory exhausted while postgres 'eats'
almost 113 Mb of RAM. This happens with pgsql 6.3.2+patches plus latest
Vadim's one on Linux 2.0.35 libc5. Sorry for variable names :-)
I remember discussion prior 6.3.2 about optimization of join and
tried various -f options but this doesn't helps.
When I run postmaster in debug mode I noticed some complains about
geqo config file doesn't exists but not sure if this important.
When I tried explain with less joins I got very bad plans with no
indices used.  Tables are not big - about 1000 records  - just
a classificators (2 columns) and only work_flats has 31 columns.
Most columns have int2, int4 types. So database looks very simple and
I didn't expect any problem.
Of course there are a bunch of indices and I did vacuum before query.
Currently I have to run a bunch of queries and do joins
by hand (using perl) :-(

Does development version 6.4 has some improvements ? I tried
latest snapshot but couldn't run because of problem with initdb and
missing anonymous  cvs account.

    Regards,

        Oleg

flats=> \i test.sql
explain select
        a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
        float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
        e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
        float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
        a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
        a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
        HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
        a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
        f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
        h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
        a.rooms = 1;
FATAL 1:  palloc failure: memory exhausted


_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83


Re: [HACKERS] join problems: FATAL 1: palloc failure: memory exhausted

From
Vadim Mikheev
Date:
Oleg Bartunov wrote:
>
> explain select
>         a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
>         float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
>         e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
>         float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
>         a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
>         a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
> from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
>         HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
> where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
>         a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
>         f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
>         h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
>         a.rooms = 1;
> FATAL 1:  palloc failure: memory exhausted

Now please try the same after

SET GEQO TO 'off';

Vadim

Re: [HACKERS] join problems: FATAL 1: palloc failure: memory exhausted

From
Bruce Momjian
Date:
Try changing the geqo parameters with SET GEQO and see if that helps.
> Hi,
>
> Below is an example of my query, which returns no plan and produces
> FATAL 1:  palloc failure: memory exhausted while postgres 'eats'
> almost 113 Mb of RAM. This happens with pgsql 6.3.2+patches plus latest
> Vadim's one on Linux 2.0.35 libc5. Sorry for variable names :-)
> I remember discussion prior 6.3.2 about optimization of join and
> tried various -f options but this doesn't helps.
> When I run postmaster in debug mode I noticed some complains about
> geqo config file doesn't exists but not sure if this important.
> When I tried explain with less joins I got very bad plans with no
> indices used.  Tables are not big - about 1000 records  - just
> a classificators (2 columns) and only work_flats has 31 columns.
> Most columns have int2, int4 types. So database looks very simple and
> I didn't expect any problem.
> Of course there are a bunch of indices and I did vacuum before query.
> Currently I have to run a bunch of queries and do joins
> by hand (using perl) :-(
>
> Does development version 6.4 has some improvements ? I tried
> latest snapshot but couldn't run because of problem with initdb and
> missing anonymous  cvs account.
>
>     Regards,
>
>         Oleg
>
> flats=> \i test.sql
> explain select
>         a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
>         float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
>         e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
>         float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
>         a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
>         a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
> from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
>         HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
> where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
>         a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
>         f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
>         h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
>         a.rooms = 1;
> FATAL 1:  palloc failure: memory exhausted
>
>
> _____________________________________________________________
> Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
> Sternberg Astronomical Institute, Moscow University (Russia)
> Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
> phone: +007(095)939-16-83, +007(095)939-23-83
>
>
>


--
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Re: [HACKERS] join problems: FATAL 1: palloc failure: memory exhausted

From
Bruce Momjian
Date:
> Oleg Bartunov wrote:
> >
> > explain select
> >         a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
> >         float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
> >         e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
> >         float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
> >         a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
> >         a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
> > from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
> >         HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
> > where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
> >         a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
> >         f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
> >         h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
> >         a.rooms = 1;
> > FATAL 1:  palloc failure: memory exhausted
>
> Now please try the same after
>
> SET GEQO TO 'off';

If that fixes it, this is the second recent case where geqo was
consuming too much memory.

--
Bruce Momjian                          |  830 Blythe Avenue
maillist@candle.pha.pa.us              |  Drexel Hill, Pennsylvania 19026
  +  If your life is a hard drive,     |  (610) 353-9879(w)
  +  Christ can be your backup.        |  (610) 853-3000(h)

Re: [HACKERS] join problems: FATAL 1: palloc failure: memory exhausted

From
Oleg Bartunov
Date:
OK,
flats=> set GEQO TO 'off';
SET VARIABLE
flats=> \i test.sql
explain select
        a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
        float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
        e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
        float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
        a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
        a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
        HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
        a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
        f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
        h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
        a.rooms = 1;

waiting .... waiting .... gave up after 20 minutes :-(
It seems setting GEQO to 'off' doesn't helps.
It's on P200, 64 Mb RAM, SCSI ...

  6:23am  up 10 days, 15:14, 12 users,  load average: 1.29, 1.04, 0.65
89 processes: 86 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 50.1% user,  2.7% system,  0.0% nice, 47.2% idle
Mem:   62988K av,  62348K used,    640K free,   6624K shrd,    180K buff
Swap: 130748K av,  94956K used,  35792K free                  4296K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
22970 postgres  12   0 94160  47M   260 R       0 50.9 77.7  13:41 postgres

Vadim, I can send you about 50 Kb compressed pg_dump of this db,
if you like to see yourself.

    Oleg



On Fri, 31 Jul 1998, Vadim Mikheev wrote:

> Date: Fri, 31 Jul 1998 08:41:00 +0800
> From: Vadim Mikheev <vadim@krs.ru>
> To: Oleg Bartunov <oleg@sai.msu.su>
> Cc: hackers@postgreSQL.org
> Subject: Re: [HACKERS] join problems: FATAL 1:  palloc failure: memory exhausted
>
> Oleg Bartunov wrote:
> >
> > explain select
> >         a.ROOMS, b.STATION, a.STREET, a.HOUSE_NO, a.FLAT_NO,
> >         float8(a.PRICE)/1000.0, a.DO_METRO, c.SPOSOB_SH, a.FLOOR,
> >         e.HOUSE_TYPE_SHORT, a.HOUSE_FLOORS, float8(a.ALL_SQR)/100.0,
> >         float8(a.LIVE_SQR)/100.0, float8(a.KITCHEN)/100.0, f.SANUZEL_SHORT,
> >         a.SANUZEL_NO,  a.BALKON, a.LODZHIA, a.HAVE_PHONE, a.HAVE_LIFT,
> >         a.HAVE_MUSOR,  g.POL_SHORT, h.SOST_SHORT, i.STATUS_SHORT, a.COMMENT
> > from  WORK_FLATS a, METRO_STATIONS b, SPOSOB c, SERIALS d,
> >         HOUSE_TYPE e, SANUZELS f, POLS g, SOSTS h, STATUS i
> > where b.STATION_ID = a.METRO_ID and c.SPOSOB_ID = a.DO_METRO_ID and
> >         a.SERIAL_ID = d.SERIAL_ID and d.HOUSE_TYPE_ID = e.HOUSE_TYPE_ID and
> >         f.SANUZEL_ID = a.SANUZEL and g.POL_ID = a.POL_ID and
> >         h.SOST_ID = a.SOST_ID     and i.STATUS_ID = a.STATUS_ID and
> >         a.rooms = 1;
> > FATAL 1:  palloc failure: memory exhausted
>
> Now please try the same after
>
> SET GEQO TO 'off';
>
> Vadim
>

_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83




Re: [HACKERS] join problems: FATAL 1: palloc failure: memory exhausted

From
Vadim Mikheev
Date:
Oleg Bartunov wrote:
>
> Vadim, I can send you about 50 Kb compressed pg_dump of this db,
> if you like to see yourself.

Ok, send it.

Vadim