Thread: Query bombed: why?

After about 25 minutes of running a query with a "where not exists
'correlated subquery'", I got a whole bunch of lines printing out: "Backend
sent D message without prior T".
Could someone give me an idea of what that means, and how to deal with it?
The query is on two tables, one having 465,266 rows with indexes on all
queried fields, the other has 70 rows with no index.  "Vacuum analyze" was
done prior to running the query.
My system administrator checked out the server, and saw nothing wrong.
I intended to send a copy of the output, but I cleared my screen by mistake:
sorry.  Please give me some indulgence, I am a beancounter, not a techie :-)

Jeff Eckermann <jeckermann@verio.net> writes:
> After about 25 minutes of running a query with a "where not exists
> 'correlated subquery'", I got a whole bunch of lines printing out: "Backend
> sent D message without prior T".
> Could someone give me an idea of what that means, and how to deal with it?

How many rows were you expecting the query to produce?  (It might be
worth redoing it as a SELECT count(*) FROM ... to find out how many it
really produced.)  My first bet is that your frontend application ran
out of memory while trying to absorb the query result.  libpq is
designed to collect the whole result before handing it back to the
application, which is nice for some things but starts to look like a bad
idea when you have a huge query result.  Also, libpq doesn't react very
gracefully to running out of memory :-( --- the symptoms you describe
sound like one likely failure mode.  (We need to fix that...)

You might be able to increase your process memory limit; otherwise,
consider using DECLARE CURSOR and FETCH to retrieve the query result
a few hundred rows at a time.

            regards, tom lane

Thanks for your reply.
I was expecting not much more than 50 rows to be returned, with an absolute
maximum of 70.
I was trying to simulate an outer join by using the "where not exists"
clause, so that I would get back my full list of 70 and be able to see the
unmatched entries...

> -----Original Message-----
> From:    Tom Lane [SMTP:tgl@sss.pgh.pa.us]
> Sent:    Tuesday, May 09, 2000 12:35 PM
> To:    Jeff Eckermann
> Cc:    pgsql-general@postgresql.org
> Subject:    Re: [GENERAL] Query bombed: why?
>
> Jeff Eckermann <jeckermann@verio.net> writes:
> > After about 25 minutes of running a query with a "where not exists
> > 'correlated subquery'", I got a whole bunch of lines printing out:
> "Backend
> > sent D message without prior T".
> > Could someone give me an idea of what that means, and how to deal with
> it?
>
> How many rows were you expecting the query to produce?  (It might be
> worth redoing it as a SELECT count(*) FROM ... to find out how many it
> really produced.)  My first bet is that your frontend application ran
> out of memory while trying to absorb the query result.  libpq is
> designed to collect the whole result before handing it back to the
> application, which is nice for some things but starts to look like a bad
> idea when you have a huge query result.  Also, libpq doesn't react very
> gracefully to running out of memory :-( --- the symptoms you describe
> sound like one likely failure mode.  (We need to fix that...)
>
> You might be able to increase your process memory limit; otherwise,
> consider using DECLARE CURSOR and FETCH to retrieve the query result
> a few hundred rows at a time.
>
>             regards, tom lane

Jeff Eckermann <jeckermann@verio.net> writes:
> I was expecting not much more than 50 rows to be returned, with an absolute
> maximum of 70.
> I was trying to simulate an outer join by using the "where not exists"
> clause, so that I would get back my full list of 70 and be able to see the
> unmatched entries...

Certainly 70 rows are not going to strain memory ;-).  My guess is that
the query didn't do what you thought, but instead produced some sort
of cross-product result...

            regards, tom lane