Thread: row-attribute in EXPLAIN-output doesn't match count(*)

row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
Hi everybody, I have table which inly has 298 rows in it, but EXPLAIN ANALYZE
shows that PG thinks it has as much as 160057 rows AFAICT from the
EXPLAIN-output.

I'm using PG-8.0.0

Here is the schema:

nbeweb=> \d onp_web_index;                                 Table "public.onp_web_index"  Column    |       Type
|                          Modifiers 
-------------+-------------------+---------------------------------------------------------------id          | integer
        | not null default  
nextval('public.onp_web_index_id_seq'::text)starturl_id | integer           | not nulllang_id     | integer           |
notnullurl_host    | character varying | not nullurl_path    | character varying | not nulltitle       | character
varying| not nullcontent     | tsvector          | not nullplain_text  | character varying | not null 
Indexes:   "onp_web_index_pkey" primary key, btree (id)   "onp_web_index_url_host_key" unique, btree (url_host,
url_path)  "onp_web_index_idx" gist (content) 
Foreign-key constraints:   "$2" FOREIGN KEY (lang_id) REFERENCES code(id)   "$1" FOREIGN KEY (starturl_id) REFERENCES
onp_web_index_starturl(id)ON  
DELETE CASCADE
I've just run a "VACUUM FULL VERBOSE" on the whole database, which gave the
following output on the relevant table(onp_web_index):

INFO:  vacuuming "public.onp_web_index"
INFO:  "onp_web_index": found 0 removable, 160057 nonremovable row versions in
206940 pages
DETAIL:  159759 dead row versions cannot be removed yet.
Nonremovable row versions range from 188 to 2036 bytes long.
There were 2205045 unused item pointers.
Total free space (including removable row versions) is 1562667708 bytes.
191561 pages are or will become empty, including 0 at the end of the table.
201809 pages containing 1562243144 free bytes are potential move destinations.
CPU 4.89s/0.88u sec elapsed 286.44 sec.
INFO:  index "onp_web_index_pkey" now contains 160057 row versions in 4562
pages
DETAIL:  0 index row versions were removed.
4072 index pages have been deleted, 3429 are currently reusable.
CPU 0.09s/0.03u sec elapsed 4.77 sec.
INFO:  index "onp_web_index_url_host_key" now contains 160057 row versions in
23508 pages
DETAIL:  0 index row versions were removed.
10589 index pages have been deleted, 9885 are currently reusable.
CPU 1.19s/0.30u sec elapsed 198.82 sec.
INFO:  index "onp_web_index_idx" now contains 160057 row versions in 52050
pages
DETAIL:  0 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 2.43s/0.53u sec elapsed 270.59 sec.
INFO:  "onp_web_index": moved 160057 row versions, truncated 206940 to 14892
pages
DETAIL:  CPU 28.25s/615.27u sec elapsed 2458.49 sec.
INFO:  index "onp_web_index_pkey" now contains 160057 row versions in 4858
pages
DETAIL:  160057 index row versions were removed.
3880 index pages have been deleted, 3237 are currently reusable.
CPU 0.15s/0.41u sec elapsed 7.93 sec.
INFO:  index "onp_web_index_url_host_key" now contains 160057 row versions in
23508 pages
DETAIL:  160057 index row versions were removed.
10492 index pages have been deleted, 9788 are currently reusable.
CPU 1.07s/1.61u sec elapsed 90.54 sec.
INFO:  index "onp_web_index_idx" now contains 160057 row versions in 55361
pages
DETAIL:  160057 index row versions were removed.
0 index pages have been deleted, 0 are currently reusable.
CPU 2.40s/0.92u sec elapsed 286.41 sec.

Here it says that it removed 160057 rows, doesn't it? But still EXPLAIN shows
the output below:

nbeweb=> EXPLAIN ANALYZE SELECT count(*) from onp_web_index;
QUERYPLAN 

--------------------------------------------------------------------------------------------------------------------------Aggregate
(cost=16892.72..16892.72 rows=1 width=0) (actual  
time=362.055..362.058 rows=1 loops=1)  ->  Seq Scan on onp_web_index  (cost=0.00..16492.57 rows=160057 width=0)
(actual time=0.043..360.926 rows=298 loops=1)Total runtime: 380.155 ms
(3 rows)

nbeweb=> REINDEX table onp_web_index;
REINDEX
nbeweb=> EXPLAIN ANALYZE SELECT count(*) from onp_web_index;
QUERYPLAN 

--------------------------------------------------------------------------------------------------------------------------Aggregate
(cost=16892.72..16892.72 rows=1 width=0) (actual  
time=229.076..229.079 rows=1 loops=1)  ->  Seq Scan on onp_web_index  (cost=0.00..16492.57 rows=160057 width=0)
(actual time=0.034..227.985 rows=298 loops=1)Total runtime: 229.157 ms
(3 rows)

nbeweb=> select count(*) from onp_web_index;count
-------  298
(1 row)


Is this normal, shall I just overlook the "rows=160057" output from EXPLAIN,
or is something wrong?
What does the line "DETAIL:  159759 dead row versions cannot be removed yet."
mean?

Thanks for any info.

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andrew Sullivan
Date:
On Thu, Apr 14, 2005 at 06:38:14PM +0200, Andreas Joseph Krogh wrote:
> Hi everybody, I have table which inly has 298 rows in it, but EXPLAIN ANALYZE 
> shows that PG thinks it has as much as 160057 rows AFAICT from the 
> EXPLAIN-output.

It does have 160057 rows:

> INFO:  vacuuming "public.onp_web_index"
> INFO:  "onp_web_index": found 0 removable, 160057 nonremovable row versions in 
> 206940 pages
> DETAIL:  159759 dead row versions cannot be removed yet.          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You no doubt have a long-running transaction keeping the rows from
being recovered.  Look for "idle in transaction" as a first guess.

> DETAIL:  160057 index row versions were removed.
> 0 index pages have been deleted, 0 are currently reusable.
> CPU 2.40s/0.92u sec elapsed 286.41 sec.
> 
> Here it says that it removed 160057 rows, doesn't it? 

No; that's the index row versions.

> Is this normal, shall I just overlook the "rows=160057" output from EXPLAIN, 
> or is something wrong?
> What does the line "DETAIL:  159759 dead row versions cannot be removed yet." 
> mean?

It means that those rows were marked invalidated after some existing
transaction started.  

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
The plural of anecdote is not data.    --Roger Brinner


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Richard Huxton
Date:
Andreas Joseph Krogh wrote:
> Hi everybody, I have table which inly has 298 rows in it, but EXPLAIN ANALYZE 
> shows that PG thinks it has as much as 160057 rows AFAICT from the 
> EXPLAIN-output.
> 

>  I've just run a "VACUUM FULL VERBOSE" on the whole database, which gave the 
> following output on the relevant table(onp_web_index):
> 
> INFO:  vacuuming "public.onp_web_index"
> INFO:  "onp_web_index": found 0 removable, 160057 nonremovable row versions in 
> 206940 pages
> DETAIL:  159759 dead row versions cannot be removed yet.

A long-lived transaction can still see these row versions, so vacuum 
can't delete them. Until that transaction is committed/rolled back PG 
has to keep the old versions of those rows available, because it might 
be able to see them.

So - do you have an application/client that has had a single transaction 
open for a long time. A very long time unless you update this table a lot.

--   Richard Huxton  Archonet Ltd


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Thursday 14 April 2005 19:12, Richard Huxton wrote:
> Andreas Joseph Krogh wrote:
> > Hi everybody, I have table which inly has 298 rows in it, but EXPLAIN
> > ANALYZE shows that PG thinks it has as much as 160057 rows AFAICT from
> > the EXPLAIN-output.
> >
> >
> >  I've just run a "VACUUM FULL VERBOSE" on the whole database, which gave
> > the following output on the relevant table(onp_web_index):
> >
> > INFO:  vacuuming "public.onp_web_index"
> > INFO:  "onp_web_index": found 0 removable, 160057 nonremovable row
> > versions in 206940 pages
> > DETAIL:  159759 dead row versions cannot be removed yet.
>
> A long-lived transaction can still see these row versions, so vacuum
> can't delete them. Until that transaction is committed/rolled back PG
> has to keep the old versions of those rows available, because it might
> be able to see them.
>
> So - do you have an application/client that has had a single transaction
> open for a long time. A very long time unless you update this table a lot.

Hm, it's a web-app, and I don't hold a transaction open for a long time on
purpose, but I have a line which says "idle in transaction" fromn "ps":

postgres  2754  1646  0 Apr06 ?        00:00:07 postgres: nbe nbeweb 127.0.0.1
idle in transaction

And by the way, I lied about the PG-version, it's 7.4.5, not 8.0.0

So, what you're suggesting is that a restart of the webapp should make vacuum
able to delete those dead rows?

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andrew Sullivan
Date:
On Thu, Apr 14, 2005 at 07:21:38PM +0200, Andreas Joseph Krogh wrote:
> 
> So, what you're suggesting is that a restart of the webapp should make vacuum 
> able to delete those dead rows?

Yes, but that'll only solve your problem for now.  You'll have the
problem again soon.  What's keeping open the transaction?

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
The fact that technology doesn't work is no bar to success in the marketplace.    --Philip Greenspun


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Thursday 14 April 2005 19:33, Andrew Sullivan wrote:
> On Thu, Apr 14, 2005 at 07:21:38PM +0200, Andreas Joseph Krogh wrote:
> > So, what you're suggesting is that a restart of the webapp should make
> > vacuum able to delete those dead rows?
>
> Yes, but that'll only solve your problem for now.  You'll have the
> problem again soon.  What's keeping open the transaction?

Don't know... All my web-apps semm to have *one* line each in "ps" which says
"dbuser dbname 127.0.0.1 idle in transaction". Those are java-web-apps which
get their connections from a connection-pool(Apache-Commons ConnectionPool),
but there is exactly *one* line pr. web-app which has the "idle in
transaction" line, even tho they have many connections open each.

Any hints on how I can find out what's keeping the connection idle in a
transaction? I realize now that I should probably ask that question on the
pgsql-jdbc-list:-)

But thanks a lot for helping me finding the problem, appreciate it!

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Richard Huxton
Date:
Andreas Joseph Krogh wrote:
> On Thursday 14 April 2005 19:12, Richard Huxton wrote:
>>
>>So - do you have an application/client that has had a single transaction
>>open for a long time. A very long time unless you update this table a lot.
> 
> 
> Hm, it's a web-app, and I don't hold a transaction open for a long time on 
> purpose, but I have a line which says "idle in transaction" fromn "ps":
> 
> postgres  2754  1646  0 Apr06 ?        00:00:07 postgres: nbe nbeweb 127.0.0.1 
> idle in transaction

Some Java systems can open a transaction and leave it there idle. That 
one's been there since April 6th.

> And by the way, I lied about the PG-version, it's 7.4.5, not 8.0.0
> 
> So, what you're suggesting is that a restart of the webapp should make vacuum 
> able to delete those dead rows?

Exactly.

--  Richard Huxton  Archonet Ltd


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andrew Sullivan
Date:
On Thu, Apr 14, 2005 at 07:54:19PM +0200, Andreas Joseph Krogh wrote:
> Any hints on how I can find out what's keeping the connection idle in a 
> transaction? I realize now that I should probably ask that question on the 
> pgsql-jdbc-list:-)

Nope.  That's a problem with your pool software.  It's no doubt
issuing "BEGIN;" as soon as it connects.  It's BAD (broken as
designed) in that case.  You need to convince it not to do that, or
else you need to go around and kill -2 such connections from time to
time.

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
I remember when computers were frustrating because they *did* exactly what 
you told them to.  That actually seems sort of quaint now.    --J.D. Baldwin


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Alvaro Herrera
Date:
On Thu, Apr 14, 2005 at 02:56:23PM -0400, Andrew Sullivan wrote:
> On Thu, Apr 14, 2005 at 07:54:19PM +0200, Andreas Joseph Krogh wrote:
> > Any hints on how I can find out what's keeping the connection idle in a 
> > transaction? I realize now that I should probably ask that question on the 
> > pgsql-jdbc-list:-)
> 
> Nope.  That's a problem with your pool software.  It's no doubt
> issuing "BEGIN;" as soon as it connects.  It's BAD (broken as
> designed) in that case.  You need to convince it not to do that, or
> else you need to go around and kill -2 such connections from time to
> time.

It is a Postgres limitation as well.  We _could_ make the server "really
start the transaction" at the point the first query is issued instead of
when the BEGIN is issued.  In fact this problem would go away if we did
that.

-- 
Alvaro Herrera (<alvherre[@]dcc.uchile.cl>)
Voy a acabar con todos los humanos / con los humanos yo acabaré
voy a acabar con todos / con todos los humanos acabaré (Bender)


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andrew Sullivan
Date:
On Thu, Apr 14, 2005 at 03:04:47PM -0400, Alvaro Herrera wrote:
> 
> It is a Postgres limitation as well.  We _could_ make the server "really
> start the transaction" at the point the first query is issued instead of
> when the BEGIN is issued.  In fact this problem would go away if we did
> that.

Yeah; I seem to remember Tom Lane suggesting such an approach at one
point.  I had the vague idea it might happen in 8.0, but for some
reason I think it didn't.  (Anyway, that's probably off topic for
this list.)

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
When my information changes, I alter my conclusions.  What do you do sir?    --attr. John Maynard Keynes


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Greg Stark
Date:
Andrew Sullivan <ajs@crankycanuck.ca> writes:

> On Thu, Apr 14, 2005 at 07:21:38PM +0200, Andreas Joseph Krogh wrote:
> > 
> > So, what you're suggesting is that a restart of the webapp should make vacuum 
> > able to delete those dead rows?
> 
> Yes, but that'll only solve your problem for now.  You'll have the
> problem again soon.  What's keeping open the transaction?

This is presumably because of the long-standing issue that Postgres takes the
snapshot as soon as the BEGIN is issued. A lot of drivers issue a "COMMIT;
BEGIN;" right away even though it could be a long time before any actual work
is done.

Other databases (by which I mean Oracle) treat BEGIN as a noop. The snapshot
starts when the first SQL statement that needs a snapshot is executed. So
until a SELECT is issued the connection doesn't participate in any
transactional issues like keeping old versions of records around.

-- 
greg



Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Thursday 14 April 2005 21:08, Andrew Sullivan wrote:
> On Thu, Apr 14, 2005 at 03:04:47PM -0400, Alvaro Herrera wrote:
> > It is a Postgres limitation as well.  We _could_ make the server "really
> > start the transaction" at the point the first query is issued instead of
> > when the BEGIN is issued.  In fact this problem would go away if we did
> > that.
>
> Yeah; I seem to remember Tom Lane suggesting such an approach at one
> point.  I had the vague idea it might happen in 8.0, but for some
> reason I think it didn't.  (Anyway, that's probably off topic for
> this list.)

I don't see this behaviour under 8.0.0, and it's presumably fixed:
http://archives.postgresql.org/pgsql-jdbc/2004-06/msg00018.php

Is it safe to issue a "kill -2 <pid-of-idle-process>", will the
connection-pool reconnect?

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andrew Sullivan
Date:
On Thu, Apr 14, 2005 at 09:57:03PM +0200, Andreas Joseph Krogh wrote:
> Is it safe to issue a "kill -2 <pid-of-idle-process>", will the 
> connection-pool reconnect?

The kill -2 will definitely work to shut the connectiond down
cleanly.  Actually, I should put that more precisely: you should send
SIGINT.  I don't know of any systems where that's not 2, but I
suppose it's logically possible.

As for the reconnection, it'll depend entirely on what your pool does
when its connections are closed.  You'll have to test it.

A

-- 
Andrew Sullivan  | ajs@crankycanuck.ca
Information security isn't a technological problem.  It's an economics
problem.    --Bruce Schneier


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Thursday 14 April 2005 22:44, Andrew Sullivan wrote:
> On Thu, Apr 14, 2005 at 09:57:03PM +0200, Andreas Joseph Krogh wrote:
> > Is it safe to issue a "kill -2 <pid-of-idle-process>", will the
> > connection-pool reconnect?
>
> The kill -2 will definitely work to shut the connectiond down
> cleanly.  Actually, I should put that more precisely: you should send
> SIGINT.  I don't know of any systems where that's not 2, but I
> suppose it's logically possible.
>
> As for the reconnection, it'll depend entirely on what your pool does
> when its connections are closed.  You'll have to test it.

Just did, and SIGINT didn't help(or I was too impatient), so I sent a SIGTERM,
then it wen away. But - it didn't help on my "vacuum dead rows" problem,
vacuum still won't reclaim those > 100.000 "dead" rows, not even after a
web-app restart(shut down Tomcat, ran "vacuum full verbose" then started
Tomcat again).

INFO:  vacuuming "public.onp_web_index"
INFO:  "onp_web_index": found 0 removable, 160057 nonremovable row versions in
14892 pages
DETAIL:  159759 dead row versions cannot be removed yet.
...

Any hints?

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Tom Lane
Date:
Andreas Joseph Krogh <andreak@officenet.no> writes:
> On Thursday 14 April 2005 21:08, Andrew Sullivan wrote:
>> On Thu, Apr 14, 2005 at 03:04:47PM -0400, Alvaro Herrera wrote:
>>> It is a Postgres limitation as well.  We _could_ make the server "really
>>> start the transaction" at the point the first query is issued instead of
>>> when the BEGIN is issued.  In fact this problem would go away if we did
>>> that.

> I don't see this behaviour under 8.0.0, and it's presumably fixed:
> http://archives.postgresql.org/pgsql-jdbc/2004-06/msg00018.php

Well, recent JDBC drivers avoid the problem by postponing BEGIN,
but the issue still exists for some other client software.  It's
something that we probably should fix on the server side.

IIRC the thing that was unresolved when we last discussed this
was what time now() should reflect --- when you issue BEGIN,
or when things really start to happen?  To make the change fully
transparent we'd have to define now() as "when you issued BEGIN",
but there's a pretty good argument that now() should correspond
to the time of the transaction snapshot.
        regards, tom lane


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Tom Lane
Date:
Greg Stark <gsstark@mit.edu> writes:
> This is presumably because of the long-standing issue that Postgres takes the
> snapshot as soon as the BEGIN is issued.

No, we don't set the snapshot until the first DML query is issued.  This
is critical for serializable transactions: you have to be able to take
locks before the snapshot is frozen.

There are at least three interesting events involved:   1  BEGIN command issued   2  First lock taken (presumably as a
consequenceof another command)   3  Transaction snapshot taken (ditto; might be a different command)
 

We have to start the transaction no later than event #2 since there has
to be something to hold the lock.  But it'd be easy enough to decouple
this from BEGIN, and it'd be good enough to solve the "COMMIT;BEGIN"
problem.

Which of these three times do you think now() ought to correspond to?
I recall having argued that it should be event #3 since that corresponds
to the database snapshot you see.  100% backwards compatibility would
require setting now() at event #1, but will anyone weep if we change that?
        regards, tom lane


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Greg Stark
Date:
Tom Lane <tgl@sss.pgh.pa.us> writes:

> We have to start the transaction no later than event #2 since there has
> to be something to hold the lock.  But it'd be easy enough to decouple
> this from BEGIN, and it'd be good enough to solve the "COMMIT;BEGIN"
> problem.

Oh I think I finally figured out what you're saying here. 

So vacuum doesn't really know what tuples are actually visible to the
snapshots actually taken by a transaction? It's making the conservative
estimate that a snapshot could have been taken as early as the start of the
transaction even if no snapshot was taken until later?


> Which of these three times do you think now() ought to correspond to?
> I recall having argued that it should be event #3 since that corresponds
> to the database snapshot you see.  100% backwards compatibility would
> require setting now() at event #1, but will anyone weep if we change that?

I think it would be weird to perform a select and see records with dates after
now(). It would also be weird to perform a select and not see records inserted
before now(). I'm not sure any of the above guarantees those criteria for READ
COMMITTED mode, but I think I'm on the same general path as you.

-- 
greg



Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Tom Lane
Date:
Greg Stark <gsstark@mit.edu> writes:
> So vacuum doesn't really know what tuples are actually visible to the
> snapshots actually taken by a transaction? It's making the conservative
> estimate that a snapshot could have been taken as early as the start of the
> transaction even if no snapshot was taken until later?

Not quite, because what it looks at is the snapshot xmin, which each
backend publishes in the PGPROC array.  A backend that has started a
transaction but hasn't yet set its snapshot can be recognized.

IIRC the problem comes up indirectly, because such a backend affects
the xmins that are computed by *other* transactions.  What VACUUM is
actually using as the cutoff is the least xmin that it can see anywhere
in PGPROC --- and everyone else's xmin will be no higher than the XID
of the laggard, even if the laggard hasn't yet set its own xmin.

Thinking about this, it seems like xmin might not be quite the right
metric for this purpose.  It might be worth thinking about whether we
could do better with a little more info in PGPROC.
        regards, tom lane


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Markus Schaber
Date:
Hi, Andreas,

Andreas Joseph Krogh schrieb:

>>>So, what you're suggesting is that a restart of the webapp should make
>>>vacuum able to delete those dead rows?
>>Yes, but that'll only solve your problem for now.  You'll have the
>>problem again soon.  What's keeping open the transaction?
> Don't know... All my web-apps semm to have *one* line each in "ps" which says 
> "dbuser dbname 127.0.0.1 idle in transaction". Those are java-web-apps which 
> get their connections from a connection-pool(Apache-Commons ConnectionPool), 
> but there is exactly *one* line pr. web-app which has the "idle in 
> transaction" line, even tho they have many connections open each.
> Any hints on how I can find out what's keeping the connection idle in a 
> transaction? I realize now that I should probably ask that question on the 
> pgsql-jdbc-list:-)

Could you try the latest postgresql-8.0-311.jdbcX.jar? The current
versions should solve the idle in transaction problem, the previous ones
tended to issue BEGIN; just after every COMMIT, so there's always an
open transaction.

Markus

Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Markus Schaber
Date:
Hi, Andrew,

Andrew Sullivan schrieb:

> Nope.  That's a problem with your pool software.  It's no doubt
> issuing "BEGIN;" as soon as it connects.

This problem may as well be caused by some versions of the postgresql
jdbc driver, no need to blame the pool software. This is fixed with the
current build 311 (and AFAIK with 310, too).



Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Friday 15 April 2005 18:22, Markus Schaber wrote:
> Hi, Andreas,
>
> Andreas Joseph Krogh schrieb:
> >>>So, what you're suggesting is that a restart of the webapp should make
> >>>vacuum able to delete those dead rows?
> >>
> >>Yes, but that'll only solve your problem for now.  You'll have the
> >>problem again soon.  What's keeping open the transaction?
> >
> > Don't know... All my web-apps semm to have *one* line each in "ps" which
> > says "dbuser dbname 127.0.0.1 idle in transaction". Those are
> > java-web-apps which get their connections from a
> > connection-pool(Apache-Commons ConnectionPool), but there is exactly
> > *one* line pr. web-app which has the "idle in transaction" line, even tho
> > they have many connections open each. Any hints on how I can find out
> > what's keeping the connection idle in a transaction? I realize now that I
> > should probably ask that question on the pgsql-jdbc-list:-)
>
> Could you try the latest postgresql-8.0-311.jdbcX.jar? The current
> versions should solve the idle in transaction problem, the previous ones
> tended to issue BEGIN; just after every COMMIT, so there's always an
> open transaction.

I could, but is it save agains a 7.4.5 version?

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Markus Schaber
Date:
Hi, Andreas,

Andreas Joseph Krogh schrieb:

>>Could you try the latest postgresql-8.0-311.jdbcX.jar? The current
>>versions should solve the idle in transaction problem, the previous ones
>>tended to issue BEGIN; just after every COMMIT, so there's always an
>>open transaction.
> I could, but is it save agains a 7.4.5 version?

Yes, it should be. I even tested 8.0-310 against a 7.2 server, and the
other way round, without any problems.

Markus



Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Andreas Joseph Krogh
Date:
On Friday 15 April 2005 18:34, Markus Schaber wrote:
> Hi, Andreas,
>
> Andreas Joseph Krogh schrieb:
> >>Could you try the latest postgresql-8.0-311.jdbcX.jar? The current
> >>versions should solve the idle in transaction problem, the previous ones
> >>tended to issue BEGIN; just after every COMMIT, so there's always an
> >>open transaction.
> >
> > I could, but is it save agains a 7.4.5 version?
>
> Yes, it should be. I even tested 8.0-310 against a 7.2 server, and the
> other way round, without any problems.

Great, I will, thanks!

--
Andreas Joseph Krogh <andreak@officenet.no>
Senior Software Developer / Manager
gpg public_key: http://dev.officenet.no/~andreak/public_key.asc
------------------------+---------------------------------------------+
OfficeNet AS            | The most difficult thing in the world is to |
Hoffsveien 17           | know how to do a thing and to watch         |
PO. Box 425 Skøyen      | somebody else doing it wrong, without       |
0213 Oslo               | comment.                                    |
NORWAY                  |                                             |
Phone : +47 22 13 01 00 |                                             |
Direct: +47 22 13 10 03 |                                             |
Mobile: +47 909  56 963 |                                             |
------------------------+---------------------------------------------+


Re: row-attribute in EXPLAIN-output doesn't match count(*)

From
Carlos Moreno
Date:
Alvaro Herrera wrote:

> It is a Postgres limitation as well.  We _could_ make the server "really
> start the transaction" at the point the first query is issued instead of
> when the BEGIN is issued. 

And also, really finish the transaction right after the last
statement is executed, instead of waiting until the COMMIT
is issued  :-)

Carlos
--