Thread: BUG #8656: Duplicate data violating unique constraints
The following bug has been logged on the website:
Bug reference: 8656
Logged by: Maciek Sakrejda
Email address: maciek@heroku.com
PostgreSQL version: 9.3.1
Operating system: Ubuntu 12.04 LTS 64-bit
Description:
A customer has run into an issue where data in a single table was apparently
duplicated somehow, violating the unique constraint imposed by the primary
key:
=> select id, count(*) from post group by id having count(*) > 1;
id | count
---------+-------
1836573 | 2
1855409 | 2
1855421 | 2
1855634 | 2
1855665 | 2
1866749 | 2
1877913 | 2
1877914 | 2
1886591 | 2
1897572 | 2
(10 rows)
What's more, the table has 12 columns, and the data for each column for all
of these duplicate rows is identical between the two versions. This table
also has an updated_at column maintained by triggers, and the updated_at for
all of these duplicated rows seems to have happened in a brief window
between 2013-12-01 06:09:31.138317+00 and 2013-12-01 06:13:07.398258+00. As
far as we can tell, nothing unusual was happening in the application around
this time. In case the table schema itself is relevant, here it is (somewhat
anonymized at the customer's request):
Column | Type |
Modifiers
----------------------+--------------------------+------------------------------------------------------------
id | integer | not null default
nextval('post_id_seq'::regclass)
col2 | integer | not null
col3 | timestamp with time zone | not null
col4 | character varying(100) | not null
col5 | integer | not null
col6 | integer | not null
col7 | timestamp with time zone | not null
updated_at | timestamp with time zone | not null
col9 | timestamp with time zone | not null
col10 | integer | not null
col11 | integer | not null
col12 | character varying(100) | not null
Indexes:
"post_pkey" PRIMARY KEY, btree (id)
"post_col4_idx" UNIQUE CONSTRAINT, btree (col4)
"post_col2_col3_idx" btree (col2, col3) CLUSTER
Foreign-key constraints:
"post_col2_fkey" FOREIGN KEY (col2) REFERENCES other_table(id) ON DELETE
CASCADE
Triggers:
t1 BEFORE INSERT ON post FOR EACH ROW EXECUTE PROCEDURE on_insert()
t2 BEFORE UPDATE ON post FOR EACH ROW EXECUTE PROCEDURE on_update()
CREATE OR REPLACE FUNCTION on_insert()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.version = 1;
NEW.created_at = timezone('UTC', now());
NEW.updated_at = NEW.created_at;
RETURN NEW;
END;
$function$
CREATE OR REPLACE FUNCTION on_update()
RETURNS trigger
LANGUAGE plpgsql
AS $function$
BEGIN
NEW.version = OLD.version + 1;
NEW.created_at = OLD.created_at;
NEW.updated_at = timezone('UTC', now());
RETURN NEW;
END;
$function$
The data has been deleted in the primary system, but it was captured in a
pg_dump backup, and I was able to restore that (except for the unique
constraints, obviously) in a separate database and can dig in further.
This is a fresh database, restored from a pg_dump backup this past Saturday,
*not* a promoted replica, so I don't think the recent replication issues
come into play here.
Any ideas?
On 2013-12-04 20:04:15 +0000, maciek@heroku.com wrote: > Bug reference: 8656 > Logged by: Maciek Sakrejda > Email address: maciek@heroku.com > PostgreSQL version: 9.3.1 > Operating system: Ubuntu 12.04 LTS 64-bit > Description: > > A customer has run into an issue where data in a single table was apparently > duplicated somehow, violating the unique constraint imposed by the primary > key: This quite possibly is an instance of the second item mentioned on http://www.postgresql.org/docs/devel/static/release-9-3-2.html (upcoming release). Note that, due to time constraints, the fix isn't perfect, but it will prevent that sort of issue. Did you ever do a VACUUM FREEZE? Do you have significant concurrency? Could you please send the output of pg_controldata and SELECT relfrozenxid, relminmxid FROM pg_class WHERE oid = 'tablename'::regclass;? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
On 2013-12-04 21:29:21 -0800, Maciek Sakrejda wrote:
> Interesting. That very well could be the issue. I didn't this affected
> primaries.
Well, it's a different, 9.3 only,bug, to the replication one.
> Latest checkpoint's NextXID: 0/1579944
> Latest checkpoint's NextMultiXactId: 592631
>
> From the description, I can't tell what this means: does this either confim
> or refute that this database was affected?
It doesn't say anything definitive either way. Do you run with any
non-default settings? Possibly vacuum_freeze_min_age,
vacuum_freeze_table_age, autovacuum_freeze_max_age?
Could you do:
SELECT ctid FROM post WHERE rows_re_affected;
That will spin out a bunch of rows like '(page,item)'.
With that page could you do:
CREATE EXTENSION pageinspect;
SELECT * FROM page_header(get_raw_page('post', page_from_above));
SELECT * FROM heap_page_items(get_raw_page('post', page_from_above));
and post that here?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Interesting. That very well could be the issue. I didn't this affected
primaries.
I don't think the customer has done a VACUUM FREEZE; I'll double-check. It
looks like there were around 20 backends and 200 commits per second around
that time based on our monitoring.
=> SELECT relfrozenxid, relminmxid FROM pg_class WHERE oid =
'post'::regclass;
relfrozenxid | relminmxid
--------------+------------
17903 | 4245098283
(1 row)
$ pg_controldata $PGDATA
pg_control version number: 937
Catalog version number: 201306121
Database system identifier: 5951409877055172578
Database cluster state: in production
pg_control last modified: Thu 05 Dec 2013 05:05:13 AM UTC
Latest checkpoint location: 1D/D917DAD8
Prior checkpoint location: 1D/CF32F6C8
Latest checkpoint's REDO location: 1D/D43487E8
Latest checkpoint's REDO WAL file: 000000010000001D000000D4
Latest checkpoint's TimeLineID: 1
Latest checkpoint's PrevTimeLineID: 1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID: 0/1579944
Latest checkpoint's NextOID: 712704
Latest checkpoint's NextMultiXactId: 592631
Latest checkpoint's NextMultiOffset: 1236955
Latest checkpoint's oldestXID: 1845
Latest checkpoint's oldestXID's DB: 1
Latest checkpoint's oldestActiveXID: 1579743
Latest checkpoint's oldestMultiXid: 1
Latest checkpoint's oldestMulti's DB: 1
Time of latest checkpoint: Thu 05 Dec 2013 05:00:18 AM UTC
Fake LSN counter for unlogged rels: 0/1
Minimum recovery ending location: 0/0
Min recovery ending loc's timeline: 0
Backup start location: 0/0
Backup end location: 0/0
End-of-backup record required: no
Current wal_level setting: hot_standby
Current max_connections setting: 500
Current max_prepared_xacts setting: 0
Current max_locks_per_xact setting: 64
Maximum data alignment: 8
Database block size: 8192
Blocks per segment of large relation: 131072
WAL block size: 8192
Bytes per WAL segment: 16777216
Maximum length of identifiers: 64
Maximum columns in an index: 32
Maximum size of a TOAST chunk: 1996
Date/time type storage: 64-bit integers
Float4 argument passing: by value
Float8 argument passing: by value
Data page checksum version: 1
From the description, I can't tell what this means: does this either confim
or refute that this database was affected?
Thanks,
Maciek
On 2013-12-05 17:21:04 -0800, Maciek Sakrejda wrote:
> On Thu, Dec 5, 2013 at 1:15 AM, Andres Freund <andres@2ndquadrant.com>wrote:
> The original data has been deleted, but I did manage to do PITR to a point
> that exhibits the issue:
Cool..
> => with affected_pages as (
> select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
> from post where id in (select id from post group by id having count(*) >
> 1)
> ) select page_header(get_raw_page('post', page::integer)) from
> affected_pages;
Hm, could you rewrite that to:
...
select * FROM affectec_pages, page_header(get_raw_page('post', page::integer));
That should work in 9.3 and will give us a much more readable result.
> => with affected_pages as (
> select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
> from post where id in (select id from post group by id having count(*) >
> 1)
> ) select heap_page_items(get_raw_page('post', page::integer)) from
> affected_pages;
Same here.
Thanks!
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
On Thu, Dec 5, 2013 at 1:15 AM, Andres Freund <andres@2ndquadrant.com>wrote:
> It doesn't say anything definitive either way. Do you run with any
> non-default settings? Possibly vacuum_freeze_min_age,
> vacuum_freeze_table_age, autovacuum_freeze_max_age?
>
No, all the vacuum-related stuff is default:
=> select name, setting, source from pg_settings where source <> 'default'
and name like '%vacuum%';
name | setting | source
------+---------+--------
(0 rows)
And no, the customer hasn't run VACUUM FREEZE.
Could you do:
> SELECT ctid FROM post WHERE rows_re_affected;
> That will spin out a bunch of rows like '(page,item)'.
>
The original data has been deleted, but I did manage to do PITR to a point
that exhibits the issue:
=> with affected_pages as (
select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
from post where id in (select id from post group by id having count(*) >
1)
) select page_header(get_raw_page('post', page::integer)) from
affected_pages;
page_header
----------------------------------------------
(12/1B004250,-1751,1,304,1712,8192,8192,4,0)
(12/1D1304D8,10886,1,316,1352,8192,8192,4,0)
(12/BE2BDF8,-12970,0,284,1592,8192,8192,4,0)
(12/BE2DDE0,1235,0,284,392,8192,8192,4,0)
(12/BE33218,-7837,0,284,392,8192,8192,4,0)
(12/BE4CF70,-20573,0,284,392,8192,8192,4,0)
(12/1D135E78,-8032,1,300,992,8192,8192,4,0)
(12/BE67610,23108,0,284,872,8192,8192,4,0)
(12/1D164030,-24596,0,284,392,8192,8192,4,0)
(9 rows)
=> with affected_pages as (
select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
from post where id in (select id from post group by id having count(*) >
1)
) select heap_page_items(get_raw_page('post', page::integer)) from
affected_pages;
heap_page_items
-----------------------------------------------------------------
(1,8072,1,118,18099,0,0,"(28192,1)",12,2306,24,,)
(2,7952,1,118,18099,80873,13,"(28192,2)",12,6466,24,,)
(3,7832,1,118,18099,80873,12,"(28192,3)",12,6466,24,,)
(4,0,3,0,,,,,,,,,)
(5,0,3,0,,,,,,,,,)
(6,7712,1,117,224637,224637,12,"(28192,6)",12,10642,24,,)
(7,0,3,0,,,,,,,,,)
(8,7592,1,117,225753,209014,3,"(28192,66)",24588,14658,24,,)
(9,0,3,0,,,,,,,,,)
(10,7472,1,118,225523,208938,1,"(28192,70)",24588,14658,24,,)
(11,0,3,0,,,,,,,,,)
(12,7352,1,119,224663,208645,2,"(28192,67)",24588,14658,24,,)
(13,0,3,0,,,,,,,,,)
(14,7232,1,117,224832,224832,100,"(28192,14)",12,10642,24,,)
(15,0,3,0,,,,,,,,,)
(16,7112,1,117,224832,208723,125,"(28192,68)",24588,14658,24,,)
(17,0,3,0,,,,,,,,,)
(18,6992,1,119,224626,214180,9,"(28192,69)",24588,14658,24,,)
(19,0,3,0,,,,,,,,,)
(20,6872,1,118,224626,214180,8,"(28192,68)",24588,14658,24,,)
(21,0,3,0,,,,,,,,,)
(22,6752,1,118,225505,208929,3,"(28192,69)",24588,14658,24,,)
(23,0,3,0,,,,,,,,,)
(24,6632,1,119,232260,269002,0,"(28192,70)",24588,14658,24,,)
(25,6512,1,118,18099,0,0,"(28192,25)",12,2306,24,,)
(26,6392,1,118,18099,0,0,"(28192,26)",12,2306,24,,)
(27,6272,1,118,18099,0,0,"(28192,27)",12,2306,24,,)
(28,6152,1,118,18099,0,0,"(28192,28)",12,2306,24,,)
(29,6032,1,118,18099,0,0,"(28192,29)",12,2306,24,,)
(30,5912,1,118,18099,0,0,"(28192,30)",12,2306,24,,)
(31,5792,1,119,18099,0,0,"(28192,31)",12,2306,24,,)
(32,5672,1,118,18099,0,0,"(28192,32)",12,2306,24,,)
(33,5552,1,118,18099,0,0,"(28192,33)",12,2306,24,,)
(34,5432,1,118,18099,0,0,"(28192,34)",12,2306,24,,)
(35,5312,1,119,18099,0,0,"(28192,35)",12,2306,24,,)
(36,5192,1,118,18099,0,0,"(28192,36)",12,2306,24,,)
(37,5072,1,118,18099,0,0,"(28192,37)",12,2306,24,,)
(38,4952,1,118,18099,0,0,"(28192,38)",12,2306,24,,)
(39,4832,1,118,18099,0,0,"(28192,39)",12,2306,24,,)
(40,4712,1,118,18099,0,0,"(28192,40)",12,2306,24,,)
(41,4592,1,118,18099,0,0,"(28192,41)",12,2306,24,,)
(42,4472,1,118,18099,0,0,"(28192,42)",12,2306,24,,)
(43,4352,1,118,18099,0,0,"(28192,43)",12,2306,24,,)
(44,4232,1,118,18099,0,0,"(28192,44)",12,2306,24,,)
(45,4112,1,118,18099,0,0,"(28192,45)",12,2306,24,,)
(46,3992,1,118,18099,0,0,"(28192,46)",12,2306,24,,)
(47,3872,1,118,18099,0,0,"(28192,47)",12,2306,24,,)
(48,3752,1,118,18099,0,0,"(28192,48)",12,2306,24,,)
(49,3632,1,118,18099,0,0,"(28192,49)",12,2306,24,,)
(50,3512,1,118,18099,0,0,"(28192,50)",12,2306,24,,)
(51,3392,1,119,18099,0,0,"(28192,51)",12,2306,24,,)
(52,3272,1,118,18099,0,0,"(28192,52)",12,2306,24,,)
(53,3152,1,118,18099,0,0,"(28192,53)",12,2306,24,,)
(54,3032,1,118,18099,0,0,"(28192,54)",12,2306,24,,)
(55,2912,1,118,18099,0,0,"(28192,55)",12,2306,24,,)
(56,2792,1,119,18099,0,0,"(28192,56)",12,2306,24,,)
(57,2672,1,118,18099,0,0,"(28192,57)",12,2306,24,,)
(58,2552,1,118,18099,0,0,"(28192,58)",12,2306,24,,)
(59,2432,1,118,18099,0,0,"(28192,59)",12,2306,24,,)
(60,2312,1,118,18099,0,0,"(28192,60)",12,2306,24,,)
(61,2192,1,118,18099,0,0,"(28192,61)",12,2306,24,,)
(62,2072,1,118,18099,0,0,"(28192,62)",12,2306,24,,)
(63,1952,1,118,18099,0,0,"(28192,63)",12,2306,24,,)
(64,1832,1,118,18099,0,0,"(28192,64)",12,2306,24,,)
(65,1712,1,118,18099,0,0,"(28192,65)",12,2306,24,,)
(66,0,0,0,,,,,,,,,)
(67,0,0,0,,,,,,,,,)
(68,0,0,0,,,,,,,,,)
(69,0,0,0,,,,,,,,,)
(70,0,0,0,,,,,,,,,)
(1,8072,1,119,224626,214180,11,"(30065,2)",8204,14658,24,,)
(2,7952,1,118,224626,214180,10,"(30065,1)",8204,14658,24,,)
(3,7832,1,119,224626,214180,9,"(28194,73)",24588,14658,24,,)
(4,7712,1,118,224626,214180,8,"(28194,72)",24588,14658,24,,)
(5,7592,1,118,224626,214180,7,"(28194,71)",24588,14658,24,,)
(6,7472,1,118,224626,214180,6,"(28194,70)",24588,14658,24,,)
(7,7352,1,118,224626,214180,5,"(28194,69)",24588,14658,24,,)
(8,7232,1,118,224626,214180,4,"(28194,68)",24588,6466,24,,)
(9,7112,1,118,224626,214180,3,"(28194,67)",24588,6466,24,,)
(10,6992,1,119,224626,214180,2,"(28194,66)",24588,6466,24,,)
(11,6872,1,119,18099,0,0,"(28194,11)",12,2306,24,,)
(12,6752,1,117,18099,0,0,"(28194,12)",12,2306,24,,)
(13,6632,1,118,18099,0,0,"(28194,13)",12,2306,24,,)
(14,6512,1,118,18099,0,0,"(28194,14)",12,2306,24,,)
(15,6392,1,118,18099,0,0,"(28194,15)",12,2306,24,,)
(16,6272,1,117,18099,0,0,"(28194,16)",12,2306,24,,)
(17,6152,1,117,18099,0,0,"(28194,17)",12,2306,24,,)
(18,6032,1,118,18099,0,0,"(28194,18)",12,2306,24,,)
(19,5912,1,118,18099,0,0,"(28194,19)",12,2306,24,,)
(20,5792,1,118,18099,0,0,"(28194,20)",12,2306,24,,)
(21,5672,1,119,18099,0,0,"(28194,21)",12,2306,24,,)
(22,5552,1,118,18099,0,0,"(28194,22)",12,2306,24,,)
(23,5432,1,119,18099,0,0,"(28194,23)",12,2306,24,,)
(24,5312,1,118,18099,0,0,"(28194,24)",12,2306,24,,)
(25,5192,1,118,18099,0,0,"(28194,25)",12,2306,24,,)
(26,5072,1,118,18099,0,0,"(28194,26)",12,2306,24,,)
(27,4952,1,119,18099,0,0,"(28194,27)",12,2306,24,,)
(28,4832,1,118,18099,0,0,"(28194,28)",12,2306,24,,)
(29,4712,1,118,18099,0,0,"(28194,29)",12,2306,24,,)
(30,4592,1,119,18099,0,0,"(28194,30)",12,2306,24,,)
(31,4472,1,117,18099,0,0,"(28194,31)",12,2306,24,,)
(32,4352,1,119,18099,0,0,"(28194,32)",12,2306,24,,)
(33,4232,1,119,18099,0,0,"(28194,33)",12,2306,24,,)
(34,4112,1,118,18099,0,0,"(28194,34)",12,2306,24,,)
(35,3992,1,119,18099,0,0,"(28194,35)",12,2306,24,,)
(36,3872,1,117,18099,0,0,"(28194,36)",12,2306,24,,)
(37,3752,1,119,18099,0,0,"(28194,37)",12,2306,24,,)
(38,3632,1,118,18099,0,0,"(28194,38)",12,2306,24,,)
(39,3512,1,118,18099,0,0,"(28194,39)",12,2306,24,,)
(40,3392,1,118,18099,0,0,"(28194,40)",12,2306,24,,)
(41,0,3,0,,,,,,,,,)
(42,0,3,0,,,,,,,,,)
(43,0,3,0,,,,,,,,,)
(44,0,3,0,,,,,,,,,)
(45,0,3,0,,,,,,,,,)
(46,0,3,0,,,,,,,,,)
(47,0,3,0,,,,,,,,,)
(48,0,3,0,,,,,,,,,)
(49,3272,1,118,18099,0,0,"(28194,49)",12,2306,24,,)
(50,3152,1,118,18099,0,0,"(28194,50)",12,2306,24,,)
(51,3032,1,118,18099,0,0,"(28194,51)",12,2306,24,,)
(52,2912,1,118,18099,0,0,"(28194,52)",12,2306,24,,)
(53,2792,1,118,18099,0,0,"(28194,53)",12,2306,24,,)
(54,2672,1,118,18099,0,0,"(28194,54)",12,2306,24,,)
(55,2552,1,118,18099,0,0,"(28194,55)",12,2306,24,,)
(56,2432,1,118,18099,0,0,"(28194,56)",12,2306,24,,)
(57,2312,1,118,18099,0,0,"(28194,57)",12,2306,24,,)
(58,2192,1,118,18099,0,0,"(28194,58)",12,2306,24,,)
(59,2072,1,118,18099,0,0,"(28194,59)",12,2306,24,,)
(60,1952,1,119,18099,0,0,"(28194,60)",12,2306,24,,)
(61,1832,1,118,18099,0,0,"(28194,61)",12,2306,24,,)
(62,1712,1,118,18099,0,0,"(28194,62)",12,2306,24,,)
(63,1592,1,118,18099,0,0,"(28194,63)",12,2306,24,,)
(64,1472,1,118,18099,0,0,"(28194,64)",12,2306,24,,)
(65,1352,1,118,18099,0,0,"(28194,65)",12,2306,24,,)
(66,0,0,0,,,,,,,,,)
(67,0,0,0,,,,,,,,,)
(68,0,0,0,,,,,,,,,)
(69,0,0,0,,,,,,,,,)
(70,0,0,0,,,,,,,,,)
(71,0,0,0,,,,,,,,,)
(72,0,0,0,,,,,,,,,)
(73,0,0,0,,,,,,,,,)
(1,8072,1,117,18099,0,0,"(28203,1)",12,2306,24,,)
(2,7952,1,117,18099,0,0,"(28203,2)",12,2306,24,,)
(3,7832,1,118,18099,0,0,"(28203,3)",12,2306,24,,)
(4,7712,1,117,224637,224637,14,"(28203,4)",12,10642,24,,)
(5,7592,1,118,224637,224637,13,"(28203,5)",12,10642,24,,)
(6,7472,1,117,224637,224637,12,"(28203,6)",12,10642,24,,)
(7,7352,1,117,224637,224637,11,"(28203,7)",12,10642,24,,)
(8,7232,1,117,224637,224637,10,"(28203,8)",12,10642,24,,)
(9,0,3,0,,,,,,,,,)
(10,0,3,0,,,,,,,,,)
(11,0,3,0,,,,,,,,,)
(12,0,3,0,,,,,,,,,)
(13,0,3,0,,,,,,,,,)
(14,0,3,0,,,,,,,,,)
(15,0,3,0,,,,,,,,,)
(16,0,3,0,,,,,,,,,)
(17,0,3,0,,,,,,,,,)
(18,0,3,0,,,,,,,,,)
(19,7112,1,117,18099,0,0,"(28203,19)",12,2306,24,,)
(20,6992,1,117,18099,0,0,"(28203,20)",12,2306,24,,)
(21,6872,1,117,18099,0,0,"(28203,21)",12,2306,24,,)
(22,6752,1,118,18099,0,0,"(28203,22)",12,2306,24,,)
(23,6632,1,118,18099,0,0,"(28203,23)",12,2306,24,,)
(24,6512,1,118,18099,0,0,"(28203,24)",12,2306,24,,)
(25,6392,1,117,18099,0,0,"(28203,25)",12,2306,24,,)
(26,6272,1,117,18099,0,0,"(28203,26)",12,2306,24,,)
(27,6152,1,119,18099,0,0,"(28203,27)",12,2306,24,,)
(28,6032,1,117,18099,0,0,"(28203,28)",12,2306,24,,)
(29,5912,1,119,18099,0,0,"(28203,29)",12,2306,24,,)
(30,5792,1,117,18099,0,0,"(28203,30)",12,2306,24,,)
(31,5672,1,119,18099,0,0,"(28203,31)",12,2306,24,,)
(32,5552,1,117,18099,0,0,"(28203,32)",12,2306,24,,)
(33,5432,1,119,18099,0,0,"(28203,33)",12,2306,24,,)
(34,5312,1,117,18099,0,0,"(28203,34)",12,2306,24,,)
(35,5192,1,117,18099,0,0,"(28203,35)",12,2306,24,,)
(36,5072,1,117,18099,0,0,"(28203,36)",12,2306,24,,)
(37,4952,1,119,18099,0,0,"(28203,37)",12,2306,24,,)
(38,4832,1,117,18099,0,0,"(28203,38)",12,2306,24,,)
(39,4712,1,117,18099,0,0,"(28203,39)",12,2306,24,,)
(40,4592,1,117,18099,0,0,"(28203,40)",12,2306,24,,)
(41,4472,1,119,18099,0,0,"(28203,41)",12,2306,24,,)
(42,4352,1,117,18099,0,0,"(28203,42)",12,2306,24,,)
(43,4232,1,119,18099,0,0,"(28203,43)",12,2306,24,,)
(44,4112,1,119,18099,0,0,"(28203,44)",12,2306,24,,)
(45,3992,1,119,18099,0,0,"(28203,45)",12,2306,24,,)
(46,3872,1,118,18099,0,0,"(28203,46)",12,2306,24,,)
(47,3752,1,118,18099,0,0,"(28203,47)",12,2306,24,,)
(48,3632,1,118,18099,0,0,"(28203,48)",12,2306,24,,)
(49,3512,1,118,18099,0,0,"(28203,49)",12,2306,24,,)
(50,3392,1,118,18099,0,0,"(28203,50)",12,2306,24,,)
(51,3272,1,118,18099,0,0,"(28203,51)",12,2306,24,,)
(52,3152,1,118,18099,0,0,"(28203,52)",12,2306,24,,)
(53,3032,1,118,18099,0,0,"(28203,53)",12,2306,24,,)
(54,2912,1,118,18099,0,0,"(28203,54)",12,2306,24,,)
(55,2792,1,118,18099,0,0,"(28203,55)",12,2306,24,,)
(56,2672,1,118,18099,0,0,"(28203,56)",12,2306,24,,)
(57,2552,1,118,18099,0,0,"(28203,57)",12,2306,24,,)
(58,2432,1,117,18099,0,0,"(28203,58)",12,2306,24,,)
(59,2312,1,117,18099,0,0,"(28203,59)",12,2306,24,,)
(60,2192,1,118,18099,0,0,"(28203,60)",12,2306,24,,)
(61,2072,1,117,18099,0,0,"(28203,61)",12,2306,24,,)
(62,1952,1,119,18099,0,0,"(28203,62)",12,2306,24,,)
(63,1832,1,119,18099,0,0,"(28203,63)",12,2306,24,,)
(64,1712,1,119,18099,0,0,"(28203,64)",12,2306,24,,)
(65,1592,1,117,18099,0,0,"(28203,65)",12,2306,24,,)
(1,8072,1,117,18099,0,0,"(28204,1)",12,2306,24,,)
(2,7952,1,117,18099,0,0,"(28204,2)",12,2306,24,,)
(3,7832,1,117,18099,0,0,"(28204,3)",12,2306,24,,)
(4,7712,1,119,18099,0,0,"(28204,4)",12,2306,24,,)
(5,7592,1,119,18099,0,0,"(28204,5)",12,2306,24,,)
(6,7472,1,117,18099,0,0,"(28204,6)",12,2306,24,,)
(7,7352,1,118,18099,0,0,"(28204,7)",12,2306,24,,)
(8,7232,1,118,18099,0,0,"(28204,8)",12,2306,24,,)
(9,7112,1,119,18099,0,0,"(28204,9)",12,2306,24,,)
(10,6992,1,117,18099,0,0,"(28204,10)",12,2306,24,,)
(11,6872,1,117,18099,0,0,"(28204,11)",12,2306,24,,)
(12,6752,1,117,18099,0,0,"(28204,12)",12,2306,24,,)
(13,6632,1,117,18099,0,0,"(28204,13)",12,2306,24,,)
(14,6512,1,117,18099,0,0,"(28204,14)",12,2306,24,,)
(15,6392,1,119,18099,0,0,"(28204,15)",12,2306,24,,)
(16,6272,1,119,18099,0,0,"(28204,16)",12,2306,24,,)
(17,6152,1,119,18099,0,0,"(28204,17)",12,2306,24,,)
(18,6032,1,117,18099,0,0,"(28204,18)",12,2306,24,,)
(19,5912,1,119,18099,0,0,"(28204,19)",12,2306,24,,)
(20,5792,1,117,18099,0,0,"(28204,20)",12,2306,24,,)
(21,5672,1,117,18099,0,0,"(28204,21)",12,2306,24,,)
(22,5552,1,117,18099,0,0,"(28204,22)",12,2306,24,,)
(23,5432,1,117,18099,0,0,"(28204,23)",12,2306,24,,)
(24,5312,1,119,18099,0,0,"(28204,24)",12,2306,24,,)
(25,5192,1,117,18099,0,0,"(28204,25)",12,2306,24,,)
(26,5072,1,119,224663,224663,8,"(28204,26)",12,10642,24,,)
(27,4952,1,119,224663,224663,7,"(28204,27)",12,10642,24,,)
(28,4832,1,119,224663,224663,6,"(28204,28)",12,10642,24,,)
(29,4712,1,118,224663,224663,5,"(28204,29)",12,10642,24,,)
(30,4592,1,117,224663,224663,4,"(28204,30)",12,10642,24,,)
(31,4472,1,119,224663,224663,3,"(28204,31)",12,10642,24,,)
(32,4352,1,119,224663,208645,2,"(29998,54)",8204,12610,24,,)
(33,4232,1,119,224663,208645,1,"(29998,53)",8204,12610,24,,)
(34,4112,1,119,224663,208645,0,"(29998,52)",8204,12610,24,,)
(35,3992,1,117,18099,0,0,"(28204,35)",12,2306,24,,)
(36,3872,1,117,18099,0,0,"(28204,36)",12,2306,24,,)
(37,3752,1,117,18099,0,0,"(28204,37)",12,2306,24,,)
(38,3632,1,117,18099,0,0,"(28204,38)",12,2306,24,,)
(39,3512,1,117,18099,0,0,"(28204,39)",12,2306,24,,)
(40,3392,1,117,18099,0,0,"(28204,40)",12,2306,24,,)
(41,3272,1,117,18099,0,0,"(28204,41)",12,2306,24,,)
(42,3152,1,117,18099,0,0,"(28204,42)",12,2306,24,,)
(43,3032,1,117,18099,0,0,"(28204,43)",12,2306,24,,)
(44,2912,1,117,18099,0,0,"(28204,44)",12,2306,24,,)
(45,2792,1,117,18099,0,0,"(28204,45)",12,2306,24,,)
(46,2672,1,117,18099,0,0,"(28204,46)",12,2306,24,,)
(47,2552,1,117,18099,0,0,"(28204,47)",12,2306,24,,)
(48,2432,1,117,18099,0,0,"(28204,48)",12,2306,24,,)
(49,2312,1,117,18099,0,0,"(28204,49)",12,2306,24,,)
(50,2192,1,117,18099,0,0,"(28204,50)",12,2306,24,,)
(51,2072,1,117,18099,0,0,"(28204,51)",12,2306,24,,)
(52,1952,1,117,18099,0,0,"(28204,52)",12,2306,24,,)
(53,1832,1,119,18099,0,0,"(28204,53)",12,2306,24,,)
(54,1712,1,119,18099,0,0,"(28204,54)",12,2306,24,,)
(55,1592,1,117,18099,0,0,"(28204,55)",12,2306,24,,)
(56,1472,1,117,18099,0,0,"(28204,56)",12,2306,24,,)
(57,1352,1,117,18099,0,0,"(28204,57)",12,2306,24,,)
(58,1232,1,117,18099,0,0,"(28204,58)",12,2306,24,,)
(59,1112,1,117,18099,0,0,"(28204,59)",12,2306,24,,)
(60,992,1,117,18099,0,0,"(28204,60)",12,2306,24,,)
(61,872,1,117,18099,0,0,"(28204,61)",12,2306,24,,)
(62,752,1,117,18099,0,0,"(28204,62)",12,2306,24,,)
(63,632,1,117,18099,0,0,"(28204,63)",12,2306,24,,)
(64,512,1,117,18099,0,0,"(28204,64)",12,2306,24,,)
(65,392,1,117,18099,0,0,"(28204,65)",12,2306,24,,)
(1,8072,1,117,224832,224832,105,"(28281,1)",12,10642,24,,)
(2,7952,1,117,224832,224832,104,"(28281,2)",12,10642,24,,)
(3,7832,1,119,224832,224832,103,"(28281,3)",12,10642,24,,)
(4,7712,1,117,224832,224832,102,"(28281,4)",12,10642,24,,)
(5,7592,1,117,224832,224832,101,"(28281,5)",12,10642,24,,)
(6,7472,1,117,224832,224832,100,"(28281,6)",12,10642,24,,)
(7,7352,1,117,224832,224832,99,"(28281,7)",12,10642,24,,)
(8,7232,1,117,224832,224832,98,"(28281,8)",12,10642,24,,)
(9,7112,1,117,224832,224832,97,"(28281,9)",12,10642,24,,)
(10,6992,1,119,224832,224832,96,"(28281,10)",12,10642,24,,)
(11,6872,1,117,224832,224832,95,"(28281,11)",12,10642,24,,)
(12,6752,1,117,224832,224832,94,"(28281,12)",12,10642,24,,)
(13,6632,1,117,224832,224832,93,"(28281,13)",12,10642,24,,)
(14,6512,1,117,224832,224832,92,"(28281,14)",12,10642,24,,)
(15,6392,1,119,224832,224832,91,"(28281,15)",12,10642,24,,)
(16,6272,1,117,224832,224832,90,"(28281,16)",12,10642,24,,)
(17,6152,1,117,224832,224832,89,"(28281,17)",12,10642,24,,)
(18,6032,1,117,224832,224832,88,"(28281,18)",12,10642,24,,)
(19,5912,1,119,224832,224832,87,"(28281,19)",12,10642,24,,)
(20,5792,1,117,224832,224832,86,"(28281,20)",12,10642,24,,)
(21,5672,1,117,224832,224832,85,"(28281,21)",12,10642,24,,)
(22,5552,1,117,224832,224832,84,"(28281,22)",12,10642,24,,)
(23,5432,1,117,224832,224832,83,"(28281,23)",12,10642,24,,)
(24,5312,1,117,224832,224832,82,"(28281,24)",12,10642,24,,)
(25,5192,1,117,224832,224832,81,"(28281,25)",12,10642,24,,)
(26,5072,1,117,224832,224832,80,"(28281,26)",12,10642,24,,)
(27,4952,1,117,224832,224832,79,"(28281,27)",12,10642,24,,)
(28,4832,1,117,224832,224832,78,"(28281,28)",12,10642,24,,)
(29,4712,1,117,224832,224832,77,"(28281,29)",12,10642,24,,)
(30,4592,1,117,224832,208723,125,"(30008,58)",8204,12610,24,,)
(31,4472,1,117,224832,208723,124,"(30008,57)",8204,12610,24,,)
(32,4352,1,117,224832,208723,123,"(30008,56)",8204,12610,24,,)
(33,4232,1,117,224832,208723,122,"(30008,55)",8204,12610,24,,)
(34,4112,1,117,224832,208723,121,"(30008,54)",8204,12610,24,,)
(35,3992,1,119,224832,208723,120,"(30008,53)",8204,12610,24,,)
(36,3872,1,117,224832,208723,119,"(30008,52)",8204,12610,24,,)
(37,3752,1,117,224832,208723,118,"(30008,51)",8204,12610,24,,)
(38,3632,1,118,224832,208723,117,"(30008,50)",8204,12610,24,,)
(39,3512,1,117,224832,208723,116,"(30008,49)",8204,12610,24,,)
(40,3392,1,117,224832,208723,115,"(30008,48)",8204,12610,24,,)
(41,3272,1,117,224832,208723,114,"(30008,47)",8204,12610,24,,)
(42,3152,1,117,224832,208723,113,"(30008,46)",8204,12610,24,,)
(43,3032,1,117,224832,208723,112,"(30008,45)",8204,12610,24,,)
(44,2912,1,117,224832,208723,111,"(30008,44)",8204,12610,24,,)
(45,2792,1,117,224832,208723,110,"(30008,43)",8204,12610,24,,)
(46,2672,1,117,224832,208723,109,"(30008,42)",8204,12610,24,,)
(47,2552,1,117,224832,208723,108,"(30008,41)",8204,12610,24,,)
(48,2432,1,117,224832,208723,107,"(30008,40)",8204,12610,24,,)
(49,2312,1,117,224832,208723,106,"(30008,39)",8204,12610,24,,)
(50,2192,1,117,224832,208723,105,"(30008,38)",8204,12610,24,,)
(51,2072,1,117,224832,208723,104,"(30008,37)",8204,12610,24,,)
(52,1952,1,117,224832,208723,103,"(30008,36)",8204,12610,24,,)
(53,1832,1,117,224832,208723,102,"(30008,35)",8204,12610,24,,)
(54,1712,1,117,224832,208723,101,"(30008,34)",8204,12610,24,,)
(55,1592,1,119,224832,208723,100,"(30008,33)",8204,12610,24,,)
(56,1472,1,117,224832,208723,99,"(30008,32)",8204,12610,24,,)
(57,1352,1,117,224832,208723,98,"(30008,31)",8204,12610,24,,)
(58,1232,1,117,224832,208723,97,"(30008,30)",8204,12610,24,,)
(59,1112,1,119,224832,208723,96,"(30008,29)",8204,12610,24,,)
(60,992,1,117,224832,208723,95,"(30008,28)",8204,12610,24,,)
(61,872,1,117,224832,208723,94,"(30008,27)",8204,12610,24,,)
(62,752,1,117,224832,208723,93,"(30008,26)",8204,12610,24,,)
(63,632,1,117,224832,208723,92,"(30008,25)",8204,12610,24,,)
(64,512,1,117,224832,208723,91,"(30008,24)",8204,12610,24,,)
(65,392,1,117,224832,208723,90,"(30008,23)",8204,12610,24,,)
(1,8072,1,118,18099,0,0,"(28390,1)",12,2306,24,,)
(2,7952,1,118,18099,0,0,"(28390,2)",12,2306,24,,)
(3,7832,1,118,18099,0,0,"(28390,3)",12,2306,24,,)
(4,7712,1,118,18099,0,0,"(28390,4)",12,2306,24,,)
(5,7592,1,118,18099,0,0,"(28390,5)",12,2306,24,,)
(6,7472,1,118,18099,0,0,"(28390,6)",12,2306,24,,)
(7,7352,1,118,18099,0,0,"(28390,7)",12,2306,24,,)
(8,7232,1,118,18099,0,0,"(28390,8)",12,2306,24,,)
(9,7112,1,118,18099,0,0,"(28390,9)",12,2306,24,,)
(10,6992,1,118,18099,0,0,"(28390,10)",12,2306,24,,)
(11,6872,1,119,18099,0,0,"(28390,11)",12,2306,24,,)
(12,6752,1,118,225505,225505,10,"(28390,12)",12,10642,24,,)
(13,6632,1,118,225505,225505,9,"(28390,13)",12,10642,24,,)
(14,6512,1,118,225505,225505,8,"(28390,14)",12,10642,24,,)
(15,6392,1,118,225505,225505,7,"(28390,15)",12,10642,24,,)
(16,6272,1,118,225505,225505,6,"(28390,16)",12,10642,24,,)
(17,6152,1,118,225505,225505,5,"(28390,17)",12,10642,24,,)
(18,6032,1,119,225505,225505,4,"(28390,18)",12,10642,24,,)
(19,5912,1,118,225505,208929,3,"(29994,36)",8204,12610,24,,)
(20,5792,1,118,225505,208929,2,"(29994,35)",8204,12610,24,,)
(21,5672,1,118,225505,208929,1,"(29994,34)",8204,12610,24,,)
(22,5552,1,118,225505,208929,0,"(29994,33)",8204,12610,24,,)
(23,5432,1,118,18099,0,0,"(28390,23)",12,2306,24,,)
(24,5312,1,118,18099,0,0,"(28390,24)",12,2306,24,,)
(25,5192,1,118,18099,0,0,"(28390,25)",12,2306,24,,)
(26,5072,1,118,18099,0,0,"(28390,26)",12,2306,24,,)
(27,4952,1,118,18099,0,0,"(28390,27)",12,2306,24,,)
(28,4832,1,118,18099,0,0,"(28390,28)",12,2306,24,,)
(29,4712,1,118,18099,0,0,"(28390,29)",12,2306,24,,)
(30,4592,1,118,18099,0,0,"(28390,30)",12,2306,24,,)
(31,4472,1,118,18099,0,0,"(28390,31)",12,2306,24,,)
(32,4352,1,118,18099,0,0,"(28390,32)",12,2306,24,,)
(33,4232,1,117,18099,0,0,"(28390,33)",12,2306,24,,)
(34,4112,1,118,18099,0,0,"(28390,34)",12,2306,24,,)
(35,3992,1,118,18099,0,0,"(28390,35)",12,2306,24,,)
(36,3872,1,118,18099,0,0,"(28390,36)",12,2306,24,,)
(37,3752,1,119,18099,0,0,"(28390,37)",12,2306,24,,)
(38,3632,1,118,18099,0,0,"(28390,38)",12,2306,24,,)
(39,3512,1,117,18099,0,0,"(28390,39)",12,2306,24,,)
(40,3392,1,118,18099,0,0,"(28390,40)",12,2306,24,,)
(41,3272,1,118,18099,0,0,"(28390,41)",12,2306,24,,)
(42,3152,1,117,18099,0,0,"(28390,42)",12,2306,24,,)
(43,3032,1,117,18099,0,0,"(28390,43)",12,2306,24,,)
(44,2912,1,118,18099,0,0,"(28390,44)",12,2306,24,,)
(45,2792,1,118,18099,0,0,"(28390,45)",12,2306,24,,)
(46,2672,1,118,18099,0,0,"(28390,46)",12,2306,24,,)
(47,2552,1,119,18099,0,0,"(28390,47)",12,2306,24,,)
(48,2432,1,117,18099,0,0,"(28390,48)",12,2306,24,,)
(49,2312,1,118,18099,0,0,"(28390,49)",12,2306,24,,)
(50,2192,1,117,18099,0,0,"(28390,50)",12,2306,24,,)
(51,2072,1,118,18099,0,0,"(28390,51)",12,2306,24,,)
(52,1952,1,118,18099,0,0,"(28390,52)",12,2306,24,,)
(53,1832,1,117,18099,0,0,"(28390,53)",12,2306,24,,)
(54,1712,1,118,18099,0,0,"(28390,54)",12,2306,24,,)
(55,1592,1,118,18099,0,0,"(28390,55)",12,2306,24,,)
(56,1472,1,118,18099,0,0,"(28390,56)",12,2306,24,,)
(57,1352,1,118,18099,0,0,"(28390,57)",12,2306,24,,)
(58,1232,1,119,18099,0,0,"(28390,58)",12,2306,24,,)
(59,1112,1,117,18099,0,0,"(28390,59)",12,2306,24,,)
(60,992,1,118,18099,0,0,"(28390,60)",12,2306,24,,)
(61,872,1,118,18099,0,0,"(28390,61)",12,2306,24,,)
(62,752,1,118,18099,0,0,"(28390,62)",12,2306,24,,)
(63,632,1,118,18099,0,0,"(28390,63)",12,2306,24,,)
(64,512,1,117,18099,0,0,"(28390,64)",12,2306,24,,)
(65,392,1,118,18099,0,0,"(28390,65)",12,2306,24,,)
(1,0,3,0,,,,,,,,,)
(2,0,3,0,,,,,,,,,)
(3,0,3,0,,,,,,,,,)
(4,8072,1,118,18099,0,0,"(28392,4)",12,2306,24,,)
(5,0,3,0,,,,,,,,,)
(6,7952,1,118,18099,0,0,"(28392,6)",12,2306,24,,)
(7,0,3,0,,,,,,,,,)
(8,7832,1,119,18099,0,0,"(28392,8)",12,2306,24,,)
(9,7712,1,119,18099,0,0,"(28392,9)",12,2306,24,,)
(10,7592,1,117,18099,0,0,"(28392,10)",12,2306,24,,)
(11,7472,1,117,18099,0,0,"(28392,11)",12,2306,24,,)
(12,7352,1,117,18099,0,0,"(28392,12)",12,2306,24,,)
(13,7232,1,117,18099,0,0,"(28392,13)",12,2306,24,,)
(14,7112,1,119,18099,0,0,"(28392,14)",12,2306,24,,)
(15,6992,1,118,18099,0,0,"(28392,15)",12,2306,24,,)
(16,6872,1,118,18099,0,0,"(28392,16)",12,2306,24,,)
(17,6752,1,117,18099,0,0,"(28392,17)",12,2306,24,,)
(18,6632,1,117,18099,0,0,"(28392,18)",12,2306,24,,)
(19,6512,1,117,18099,0,0,"(28392,19)",12,2306,24,,)
(20,6392,1,117,18099,0,0,"(28392,20)",12,2306,24,,)
(21,6272,1,117,18099,0,0,"(28392,21)",12,2306,24,,)
(22,6152,1,117,18099,0,0,"(28392,22)",12,2306,24,,)
(23,6032,1,117,18099,0,0,"(28392,23)",12,2306,24,,)
(24,5912,1,117,18099,0,0,"(28392,24)",12,2306,24,,)
(25,66,2,0,,,,,,,,,)
(26,5792,1,117,18099,0,0,"(28392,26)",12,2306,24,,)
(27,5672,1,117,18099,0,0,"(28392,27)",12,2306,24,,)
(28,5552,1,117,18099,0,0,"(28392,28)",12,2306,24,,)
(29,5432,1,117,18099,0,0,"(28392,29)",12,2306,24,,)
(30,5312,1,117,18099,0,0,"(28392,30)",12,2306,24,,)
(31,5192,1,119,18099,0,0,"(28392,31)",12,2306,24,,)
(32,5072,1,117,18099,0,0,"(28392,32)",12,2306,24,,)
(33,4952,1,119,18099,0,0,"(28392,33)",12,2306,24,,)
(34,4832,1,117,18099,0,0,"(28392,34)",12,2306,24,,)
(35,4712,1,117,18099,0,0,"(28392,35)",12,2306,24,,)
(36,4592,1,117,18099,0,0,"(28392,36)",12,2306,24,,)
(37,4472,1,117,18099,0,0,"(28392,37)",12,2306,24,,)
(38,4352,1,118,18099,0,0,"(28392,38)",12,2306,24,,)
(39,4232,1,117,18099,0,0,"(28392,39)",12,2306,24,,)
(40,4112,1,118,18099,0,0,"(28392,40)",12,2306,24,,)
(41,3992,1,119,18099,0,0,"(28392,41)",12,2306,24,,)
(42,3872,1,119,225523,208938,2,"(28392,69)",24588,14658,24,,)
(43,3752,1,118,225523,208938,1,"(28392,68)",24588,14658,24,,)
(44,3632,1,119,225523,208938,0,"(28392,67)",24588,14658,24,,)
(45,3512,1,118,18099,0,0,"(28392,45)",12,2306,24,,)
(46,3392,1,117,18099,0,0,"(28392,46)",12,2306,24,,)
(47,3272,1,118,18099,0,0,"(28392,47)",12,2306,24,,)
(48,3152,1,117,18099,0,0,"(28392,48)",12,2306,24,,)
(49,3032,1,117,18099,0,0,"(28392,49)",12,2306,24,,)
(50,2912,1,118,18099,0,0,"(28392,50)",12,2306,24,,)
(51,2792,1,117,18099,0,0,"(28392,51)",12,2306,24,,)
(52,2672,1,119,18099,0,0,"(28392,52)",12,2306,24,,)
(53,2552,1,117,18099,0,0,"(28392,53)",12,2306,24,,)
(54,2432,1,117,18099,0,0,"(28392,54)",12,2306,24,,)
(55,2312,1,117,18099,0,0,"(28392,55)",12,2306,24,,)
(56,2192,1,117,18099,0,0,"(28392,56)",12,2306,24,,)
(57,2072,1,117,18099,0,0,"(28392,57)",12,2306,24,,)
(58,1952,1,119,18099,0,0,"(28392,58)",12,2306,24,,)
(59,1832,1,117,18099,0,0,"(28392,59)",12,2306,24,,)
(60,1712,1,118,18099,0,0,"(28392,60)",12,2306,24,,)
(61,1592,1,117,18099,0,0,"(28392,61)",12,2306,24,,)
(62,1472,1,118,18099,0,0,"(28392,62)",12,2306,24,,)
(63,1352,1,117,18099,0,0,"(28392,63)",12,2306,24,,)
(64,1232,1,117,18099,0,0,"(28392,64)",12,2306,24,,)
(65,1112,1,117,18099,0,0,"(28392,65)",12,2306,24,,)
(66,992,1,118,570583,570583,0,"(28392,66)",32780,10642,24,,)
(67,0,0,0,,,,,,,,,)
(68,0,0,0,,,,,,,,,)
(69,0,0,0,,,,,,,,,)
(1,8072,1,118,18099,0,0,"(28410,1)",12,2306,24,,)
(2,7952,1,117,18099,0,0,"(28410,2)",12,2306,24,,)
(3,7832,1,117,18099,0,0,"(28410,3)",12,2306,24,,)
(4,7712,1,117,18099,0,0,"(28410,4)",12,2306,24,,)
(5,7592,1,117,18099,0,0,"(28410,5)",12,2306,24,,)
(6,7472,1,119,18099,0,0,"(28410,6)",12,2306,24,,)
(7,7352,1,119,18099,0,0,"(28410,7)",12,2306,24,,)
(8,7232,1,117,18099,0,0,"(28410,8)",12,2306,24,,)
(9,7112,1,117,18099,0,0,"(28410,9)",12,2306,24,,)
(10,6992,1,117,18099,0,0,"(28410,10)",12,2306,24,,)
(11,6872,1,117,18099,0,0,"(28410,11)",12,2306,24,,)
(12,6752,1,117,18099,0,0,"(28410,12)",12,2306,24,,)
(13,6632,1,117,18099,0,0,"(28410,13)",12,2306,24,,)
(14,6512,1,117,18099,0,0,"(28410,14)",12,2306,24,,)
(15,6392,1,117,225753,225753,4,"(28410,15)",12,10642,24,,)
(16,6272,1,117,225753,209014,3,"(30024,41)",8204,12610,24,,)
(17,6152,1,117,225753,209014,2,"(30024,40)",8204,12610,24,,)
(18,6032,1,117,225753,209014,1,"(30024,39)",8204,12610,24,,)
(19,5912,1,117,225753,209014,0,"(30024,38)",8204,12610,24,,)
(20,5792,1,118,18099,0,0,"(28410,20)",12,2306,24,,)
(21,5672,1,118,18099,0,0,"(28410,21)",12,2306,24,,)
(22,5552,1,118,18099,0,0,"(28410,22)",12,2306,24,,)
(23,5432,1,118,18099,0,0,"(28410,23)",12,2306,24,,)
(24,5312,1,118,18099,0,0,"(28410,24)",12,2306,24,,)
(25,5192,1,118,18099,0,0,"(28410,25)",12,2306,24,,)
(26,5072,1,119,18099,0,0,"(28410,26)",12,2306,24,,)
(27,4952,1,118,18099,0,0,"(28410,27)",12,2306,24,,)
(28,4832,1,118,18099,0,0,"(28410,28)",12,2306,24,,)
(29,4712,1,118,18099,0,0,"(28410,29)",12,2306,24,,)
(30,4592,1,119,18099,0,0,"(28410,30)",12,2306,24,,)
(31,4472,1,118,18099,0,0,"(28410,31)",12,2306,24,,)
(32,4352,1,118,18099,0,0,"(28410,32)",12,2306,24,,)
(33,4232,1,118,18099,0,0,"(28410,33)",12,2306,24,,)
(34,4112,1,118,18099,0,0,"(28410,34)",12,2306,24,,)
(35,3992,1,117,18099,0,0,"(28410,35)",12,2306,24,,)
(36,3872,1,118,18099,0,0,"(28410,36)",12,2306,24,,)
(37,3752,1,118,18099,0,0,"(28410,37)",12,2306,24,,)
(38,3632,1,118,18099,0,0,"(28410,38)",12,2306,24,,)
(39,3512,1,117,18099,0,0,"(28410,39)",12,2306,24,,)
(40,3392,1,118,18099,0,0,"(28410,40)",12,2306,24,,)
(41,3272,1,118,18099,0,0,"(28410,41)",12,2306,24,,)
(42,3152,1,118,18099,0,0,"(28410,42)",12,2306,24,,)
(43,3032,1,118,18099,0,0,"(28410,43)",12,2306,24,,)
(44,2912,1,118,18099,0,0,"(28410,44)",12,2306,24,,)
(45,2792,1,118,18099,0,0,"(28410,45)",12,2306,24,,)
(46,2672,1,118,18099,0,0,"(28410,46)",12,2306,24,,)
(47,2552,1,118,18099,0,0,"(28410,47)",12,2306,24,,)
(48,2432,1,119,18099,0,0,"(28410,48)",12,2306,24,,)
(49,0,3,0,,,,,,,,,)
(50,0,3,0,,,,,,,,,)
(51,0,3,0,,,,,,,,,)
(52,0,3,0,,,,,,,,,)
(53,2312,1,118,18099,0,0,"(28410,53)",12,2306,24,,)
(54,2192,1,119,18099,0,0,"(28410,54)",12,2306,24,,)
(55,2072,1,118,18099,0,0,"(28410,55)",12,2306,24,,)
(56,1952,1,119,18099,0,0,"(28410,56)",12,2306,24,,)
(57,1832,1,118,18099,0,0,"(28410,57)",12,2306,24,,)
(58,1712,1,119,18099,0,0,"(28410,58)",12,2306,24,,)
(59,1592,1,118,18099,0,0,"(28410,59)",12,2306,24,,)
(60,1472,1,118,18099,0,0,"(28410,60)",12,2306,24,,)
(61,1352,1,119,18099,0,0,"(28410,61)",12,2306,24,,)
(62,1232,1,118,18099,0,0,"(28410,62)",12,2306,24,,)
(63,1112,1,119,18099,0,0,"(28410,63)",12,2306,24,,)
(64,992,1,118,18099,0,0,"(28410,64)",12,2306,24,,)
(65,872,1,118,18099,0,0,"(28410,65)",12,2306,24,,)
(1,8072,1,119,18099,0,0,"(28561,1)",12,2306,24,,)
(2,7952,1,119,18099,0,0,"(28561,2)",12,2306,24,,)
(3,7832,1,118,18099,0,0,"(28561,3)",12,2306,24,,)
(4,7712,1,118,18099,0,0,"(28561,4)",12,2306,24,,)
(5,7592,1,119,18099,0,0,"(28561,5)",12,2306,24,,)
(6,7472,1,118,18099,0,0,"(28561,6)",12,2306,24,,)
(7,7352,1,118,18099,0,0,"(28561,7)",12,2306,24,,)
(8,7232,1,119,18099,0,0,"(28561,8)",12,2306,24,,)
(9,7112,1,118,232260,232260,10,"(28561,9)",12,10642,24,,)
(10,6992,1,119,232260,232260,9,"(28561,10)",12,10642,24,,)
(11,6872,1,118,232260,232260,8,"(28561,11)",12,10642,24,,)
(12,6752,1,119,232260,232260,7,"(28561,12)",12,10642,24,,)
(13,6632,1,118,232260,232260,6,"(28561,13)",12,10642,24,,)
(14,6512,1,119,232260,232260,5,"(28561,14)",12,10642,24,,)
(15,6392,1,118,232260,232260,4,"(28561,15)",12,10642,24,,)
(16,6272,1,119,232260,269002,0,"(30200,31)",8204,14658,24,,)
(17,6152,1,118,232260,269002,0,"(30200,30)",8204,14658,24,,)
(18,6032,1,119,232260,269002,0,"(30200,29)",8204,6466,24,,)
(19,5912,1,118,232260,269002,0,"(30200,28)",8204,6466,24,,)
(20,5792,1,118,18099,0,0,"(28561,20)",12,2306,24,,)
(21,5672,1,118,18099,0,0,"(28561,21)",12,2306,24,,)
(22,5552,1,119,18099,0,0,"(28561,22)",12,2306,24,,)
(23,5432,1,118,18099,0,0,"(28561,23)",12,2306,24,,)
(24,5312,1,118,18099,0,0,"(28561,24)",12,2306,24,,)
(25,5192,1,118,18099,0,0,"(28561,25)",12,2306,24,,)
(26,5072,1,118,18099,0,0,"(28561,26)",12,2306,24,,)
(27,4952,1,118,18099,0,0,"(28561,27)",12,2306,24,,)
(28,4832,1,118,18099,0,0,"(28561,28)",12,2306,24,,)
(29,4712,1,118,18099,0,0,"(28561,29)",12,2306,24,,)
(30,4592,1,118,18099,0,0,"(28561,30)",12,2306,24,,)
(31,4472,1,118,18099,0,0,"(28561,31)",12,2306,24,,)
(32,4352,1,118,18099,0,0,"(28561,32)",12,2306,24,,)
(33,4232,1,118,18099,0,0,"(28561,33)",12,2306,24,,)
(34,4112,1,118,18099,0,0,"(28561,34)",12,2306,24,,)
(35,3992,1,119,18099,0,0,"(28561,35)",12,2306,24,,)
(36,3872,1,118,18099,0,0,"(28561,36)",12,2306,24,,)
(37,3752,1,118,18099,0,0,"(28561,37)",12,2306,24,,)
(38,3632,1,118,18099,0,0,"(28561,38)",12,2306,24,,)
(39,3512,1,119,18099,0,0,"(28561,39)",12,2306,24,,)
(40,3392,1,119,18099,0,0,"(28561,40)",12,2306,24,,)
(41,3272,1,119,18099,0,0,"(28561,41)",12,2306,24,,)
(42,3152,1,118,18099,0,0,"(28561,42)",12,2306,24,,)
(43,3032,1,118,18099,0,0,"(28561,43)",12,2306,24,,)
(44,2912,1,118,18099,0,0,"(28561,44)",12,2306,24,,)
(45,2792,1,118,18099,0,0,"(28561,45)",12,2306,24,,)
(46,2672,1,118,18099,0,0,"(28561,46)",12,2306,24,,)
(47,2552,1,118,18099,0,0,"(28561,47)",12,2306,24,,)
(48,2432,1,119,18099,0,0,"(28561,48)",12,2306,24,,)
(49,2312,1,118,18099,0,0,"(28561,49)",12,2306,24,,)
(50,2192,1,118,18099,0,0,"(28561,50)",12,2306,24,,)
(51,2072,1,119,18099,0,0,"(28561,51)",12,2306,24,,)
(52,1952,1,119,18099,0,0,"(28561,52)",12,2306,24,,)
(53,1832,1,118,18099,0,0,"(28561,53)",12,2306,24,,)
(54,1712,1,118,18099,0,0,"(28561,54)",12,2306,24,,)
(55,1592,1,119,18099,0,0,"(28561,55)",12,2306,24,,)
(56,1472,1,118,232282,232282,5,"(28561,56)",12,10642,24,,)
(57,1352,1,119,232282,232282,4,"(28561,57)",12,10642,24,,)
(58,1232,1,118,232282,232282,3,"(28561,58)",12,10642,24,,)
(59,1112,1,118,232282,232282,2,"(28561,59)",12,10642,24,,)
(60,992,1,118,232282,232282,1,"(28561,60)",12,10642,24,,)
(61,872,1,119,232282,232282,0,"(28561,61)",12,10642,24,,)
(62,752,1,118,18099,0,0,"(28561,62)",12,2306,24,,)
(63,632,1,118,18099,0,0,"(28561,63)",12,2306,24,,)
(64,512,1,118,18099,0,0,"(28561,64)",12,2306,24,,)
(65,392,1,119,18099,0,0,"(28561,65)",12,2306,24,,)
(602 rows)
Here it is, sorry about that:
=> with affected_pages as (
select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
from post where id in (select id from post group by id having count(*) >
1)
) select * from affected_pages, page_header(get_raw_page('post',
page::integer));
page | lsn | checksum | flags | lower | upper | special |
pagesize | version | prune_xid
-------+-------------+----------+-------+-------+-------+---------+----------+---------+-----------
28192 | 12/1B004250 | -1751 | 1 | 304 | 1712 | 8192 |
8192 | 4 | 0
28194 | 12/1D1304D8 | 10886 | 1 | 316 | 1352 | 8192 |
8192 | 4 | 0
28203 | 12/BE2BDF8 | -12970 | 0 | 284 | 1592 | 8192 |
8192 | 4 | 0
28204 | 12/BE2DDE0 | 1235 | 0 | 284 | 392 | 8192 |
8192 | 4 | 0
28281 | 12/BE33218 | -7837 | 0 | 284 | 392 | 8192 |
8192 | 4 | 0
28390 | 12/BE4CF70 | -20573 | 0 | 284 | 392 | 8192 |
8192 | 4 | 0
28392 | 12/1D135E78 | -8032 | 1 | 300 | 992 | 8192 |
8192 | 4 | 0
28410 | 12/BE67610 | 23108 | 0 | 284 | 872 | 8192 |
8192 | 4 | 0
28561 | 12/1D164030 | -24596 | 0 | 284 | 392 | 8192 |
8192 | 4 | 0
(9 rows)
=> with affected_pages as (
select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
from post where id in (select id from post group by id having count(*) >
1)
) select * from affected_pages, heap_page_items(get_raw_page('post',
page::integer));
page | lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 |
t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid
-------+----+--------+----------+--------+--------+--------+----------+------------+-------------+------------+--------+--------+-------
28192 | 1 | 8072 | 1 | 118 | 18099 | 0 | 0 |
(28192,1) | 12 | 2306 | 24 | |
28192 | 2 | 7952 | 1 | 118 | 18099 | 80873 | 13 |
(28192,2) | 12 | 6466 | 24 | |
28192 | 3 | 7832 | 1 | 118 | 18099 | 80873 | 12 |
(28192,3) | 12 | 6466 | 24 | |
28192 | 4 | 0 | 3 | 0 | | |
| | | | | |
28192 | 5 | 0 | 3 | 0 | | |
| | | | | |
28192 | 6 | 7712 | 1 | 117 | 224637 | 224637 | 12 |
(28192,6) | 12 | 10642 | 24 | |
28192 | 7 | 0 | 3 | 0 | | |
| | | | | |
28192 | 8 | 7592 | 1 | 117 | 225753 | 209014 | 3 |
(28192,66) | 24588 | 14658 | 24 | |
28192 | 9 | 0 | 3 | 0 | | |
| | | | | |
28192 | 10 | 7472 | 1 | 118 | 225523 | 208938 | 1 |
(28192,70) | 24588 | 14658 | 24 | |
28192 | 11 | 0 | 3 | 0 | | |
| | | | | |
28192 | 12 | 7352 | 1 | 119 | 224663 | 208645 | 2 |
(28192,67) | 24588 | 14658 | 24 | |
28192 | 13 | 0 | 3 | 0 | | |
| | | | | |
28192 | 14 | 7232 | 1 | 117 | 224832 | 224832 | 100 |
(28192,14) | 12 | 10642 | 24 | |
28192 | 15 | 0 | 3 | 0 | | |
| | | | | |
28192 | 16 | 7112 | 1 | 117 | 224832 | 208723 | 125 |
(28192,68) | 24588 | 14658 | 24 | |
28192 | 17 | 0 | 3 | 0 | | |
| | | | | |
28192 | 18 | 6992 | 1 | 119 | 224626 | 214180 | 9 |
(28192,69) | 24588 | 14658 | 24 | |
28192 | 19 | 0 | 3 | 0 | | |
| | | | | |
28192 | 20 | 6872 | 1 | 118 | 224626 | 214180 | 8 |
(28192,68) | 24588 | 14658 | 24 | |
28192 | 21 | 0 | 3 | 0 | | |
| | | | | |
28192 | 22 | 6752 | 1 | 118 | 225505 | 208929 | 3 |
(28192,69) | 24588 | 14658 | 24 | |
28192 | 23 | 0 | 3 | 0 | | |
| | | | | |
28192 | 24 | 6632 | 1 | 119 | 232260 | 269002 | 0 |
(28192,70) | 24588 | 14658 | 24 | |
28192 | 25 | 6512 | 1 | 118 | 18099 | 0 | 0 |
(28192,25) | 12 | 2306 | 24 | |
28192 | 26 | 6392 | 1 | 118 | 18099 | 0 | 0 |
(28192,26) | 12 | 2306 | 24 | |
28192 | 27 | 6272 | 1 | 118 | 18099 | 0 | 0 |
(28192,27) | 12 | 2306 | 24 | |
28192 | 28 | 6152 | 1 | 118 | 18099 | 0 | 0 |
(28192,28) | 12 | 2306 | 24 | |
28192 | 29 | 6032 | 1 | 118 | 18099 | 0 | 0 |
(28192,29) | 12 | 2306 | 24 | |
28192 | 30 | 5912 | 1 | 118 | 18099 | 0 | 0 |
(28192,30) | 12 | 2306 | 24 | |
28192 | 31 | 5792 | 1 | 119 | 18099 | 0 | 0 |
(28192,31) | 12 | 2306 | 24 | |
28192 | 32 | 5672 | 1 | 118 | 18099 | 0 | 0 |
(28192,32) | 12 | 2306 | 24 | |
28192 | 33 | 5552 | 1 | 118 | 18099 | 0 | 0 |
(28192,33) | 12 | 2306 | 24 | |
28192 | 34 | 5432 | 1 | 118 | 18099 | 0 | 0 |
(28192,34) | 12 | 2306 | 24 | |
28192 | 35 | 5312 | 1 | 119 | 18099 | 0 | 0 |
(28192,35) | 12 | 2306 | 24 | |
28192 | 36 | 5192 | 1 | 118 | 18099 | 0 | 0 |
(28192,36) | 12 | 2306 | 24 | |
28192 | 37 | 5072 | 1 | 118 | 18099 | 0 | 0 |
(28192,37) | 12 | 2306 | 24 | |
28192 | 38 | 4952 | 1 | 118 | 18099 | 0 | 0 |
(28192,38) | 12 | 2306 | 24 | |
28192 | 39 | 4832 | 1 | 118 | 18099 | 0 | 0 |
(28192,39) | 12 | 2306 | 24 | |
28192 | 40 | 4712 | 1 | 118 | 18099 | 0 | 0 |
(28192,40) | 12 | 2306 | 24 | |
28192 | 41 | 4592 | 1 | 118 | 18099 | 0 | 0 |
(28192,41) | 12 | 2306 | 24 | |
28192 | 42 | 4472 | 1 | 118 | 18099 | 0 | 0 |
(28192,42) | 12 | 2306 | 24 | |
28192 | 43 | 4352 | 1 | 118 | 18099 | 0 | 0 |
(28192,43) | 12 | 2306 | 24 | |
28192 | 44 | 4232 | 1 | 118 | 18099 | 0 | 0 |
(28192,44) | 12 | 2306 | 24 | |
28192 | 45 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28192,45) | 12 | 2306 | 24 | |
28192 | 46 | 3992 | 1 | 118 | 18099 | 0 | 0 |
(28192,46) | 12 | 2306 | 24 | |
28192 | 47 | 3872 | 1 | 118 | 18099 | 0 | 0 |
(28192,47) | 12 | 2306 | 24 | |
28192 | 48 | 3752 | 1 | 118 | 18099 | 0 | 0 |
(28192,48) | 12 | 2306 | 24 | |
28192 | 49 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28192,49) | 12 | 2306 | 24 | |
28192 | 50 | 3512 | 1 | 118 | 18099 | 0 | 0 |
(28192,50) | 12 | 2306 | 24 | |
28192 | 51 | 3392 | 1 | 119 | 18099 | 0 | 0 |
(28192,51) | 12 | 2306 | 24 | |
28192 | 52 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28192,52) | 12 | 2306 | 24 | |
28192 | 53 | 3152 | 1 | 118 | 18099 | 0 | 0 |
(28192,53) | 12 | 2306 | 24 | |
28192 | 54 | 3032 | 1 | 118 | 18099 | 0 | 0 |
(28192,54) | 12 | 2306 | 24 | |
28192 | 55 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28192,55) | 12 | 2306 | 24 | |
28192 | 56 | 2792 | 1 | 119 | 18099 | 0 | 0 |
(28192,56) | 12 | 2306 | 24 | |
28192 | 57 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28192,57) | 12 | 2306 | 24 | |
28192 | 58 | 2552 | 1 | 118 | 18099 | 0 | 0 |
(28192,58) | 12 | 2306 | 24 | |
28192 | 59 | 2432 | 1 | 118 | 18099 | 0 | 0 |
(28192,59) | 12 | 2306 | 24 | |
28192 | 60 | 2312 | 1 | 118 | 18099 | 0 | 0 |
(28192,60) | 12 | 2306 | 24 | |
28192 | 61 | 2192 | 1 | 118 | 18099 | 0 | 0 |
(28192,61) | 12 | 2306 | 24 | |
28192 | 62 | 2072 | 1 | 118 | 18099 | 0 | 0 |
(28192,62) | 12 | 2306 | 24 | |
28192 | 63 | 1952 | 1 | 118 | 18099 | 0 | 0 |
(28192,63) | 12 | 2306 | 24 | |
28192 | 64 | 1832 | 1 | 118 | 18099 | 0 | 0 |
(28192,64) | 12 | 2306 | 24 | |
28192 | 65 | 1712 | 1 | 118 | 18099 | 0 | 0 |
(28192,65) | 12 | 2306 | 24 | |
28192 | 66 | 0 | 0 | 0 | | |
| | | | | |
28192 | 67 | 0 | 0 | 0 | | |
| | | | | |
28192 | 68 | 0 | 0 | 0 | | |
| | | | | |
28192 | 69 | 0 | 0 | 0 | | |
| | | | | |
28192 | 70 | 0 | 0 | 0 | | |
| | | | | |
28194 | 1 | 8072 | 1 | 119 | 224626 | 214180 | 11 |
(30065,2) | 8204 | 14658 | 24 | |
28194 | 2 | 7952 | 1 | 118 | 224626 | 214180 | 10 |
(30065,1) | 8204 | 14658 | 24 | |
28194 | 3 | 7832 | 1 | 119 | 224626 | 214180 | 9 |
(28194,73) | 24588 | 14658 | 24 | |
28194 | 4 | 7712 | 1 | 118 | 224626 | 214180 | 8 |
(28194,72) | 24588 | 14658 | 24 | |
28194 | 5 | 7592 | 1 | 118 | 224626 | 214180 | 7 |
(28194,71) | 24588 | 14658 | 24 | |
28194 | 6 | 7472 | 1 | 118 | 224626 | 214180 | 6 |
(28194,70) | 24588 | 14658 | 24 | |
28194 | 7 | 7352 | 1 | 118 | 224626 | 214180 | 5 |
(28194,69) | 24588 | 14658 | 24 | |
28194 | 8 | 7232 | 1 | 118 | 224626 | 214180 | 4 |
(28194,68) | 24588 | 6466 | 24 | |
28194 | 9 | 7112 | 1 | 118 | 224626 | 214180 | 3 |
(28194,67) | 24588 | 6466 | 24 | |
28194 | 10 | 6992 | 1 | 119 | 224626 | 214180 | 2 |
(28194,66) | 24588 | 6466 | 24 | |
28194 | 11 | 6872 | 1 | 119 | 18099 | 0 | 0 |
(28194,11) | 12 | 2306 | 24 | |
28194 | 12 | 6752 | 1 | 117 | 18099 | 0 | 0 |
(28194,12) | 12 | 2306 | 24 | |
28194 | 13 | 6632 | 1 | 118 | 18099 | 0 | 0 |
(28194,13) | 12 | 2306 | 24 | |
28194 | 14 | 6512 | 1 | 118 | 18099 | 0 | 0 |
(28194,14) | 12 | 2306 | 24 | |
28194 | 15 | 6392 | 1 | 118 | 18099 | 0 | 0 |
(28194,15) | 12 | 2306 | 24 | |
28194 | 16 | 6272 | 1 | 117 | 18099 | 0 | 0 |
(28194,16) | 12 | 2306 | 24 | |
28194 | 17 | 6152 | 1 | 117 | 18099 | 0 | 0 |
(28194,17) | 12 | 2306 | 24 | |
28194 | 18 | 6032 | 1 | 118 | 18099 | 0 | 0 |
(28194,18) | 12 | 2306 | 24 | |
28194 | 19 | 5912 | 1 | 118 | 18099 | 0 | 0 |
(28194,19) | 12 | 2306 | 24 | |
28194 | 20 | 5792 | 1 | 118 | 18099 | 0 | 0 |
(28194,20) | 12 | 2306 | 24 | |
28194 | 21 | 5672 | 1 | 119 | 18099 | 0 | 0 |
(28194,21) | 12 | 2306 | 24 | |
28194 | 22 | 5552 | 1 | 118 | 18099 | 0 | 0 |
(28194,22) | 12 | 2306 | 24 | |
28194 | 23 | 5432 | 1 | 119 | 18099 | 0 | 0 |
(28194,23) | 12 | 2306 | 24 | |
28194 | 24 | 5312 | 1 | 118 | 18099 | 0 | 0 |
(28194,24) | 12 | 2306 | 24 | |
28194 | 25 | 5192 | 1 | 118 | 18099 | 0 | 0 |
(28194,25) | 12 | 2306 | 24 | |
28194 | 26 | 5072 | 1 | 118 | 18099 | 0 | 0 |
(28194,26) | 12 | 2306 | 24 | |
28194 | 27 | 4952 | 1 | 119 | 18099 | 0 | 0 |
(28194,27) | 12 | 2306 | 24 | |
28194 | 28 | 4832 | 1 | 118 | 18099 | 0 | 0 |
(28194,28) | 12 | 2306 | 24 | |
28194 | 29 | 4712 | 1 | 118 | 18099 | 0 | 0 |
(28194,29) | 12 | 2306 | 24 | |
28194 | 30 | 4592 | 1 | 119 | 18099 | 0 | 0 |
(28194,30) | 12 | 2306 | 24 | |
28194 | 31 | 4472 | 1 | 117 | 18099 | 0 | 0 |
(28194,31) | 12 | 2306 | 24 | |
28194 | 32 | 4352 | 1 | 119 | 18099 | 0 | 0 |
(28194,32) | 12 | 2306 | 24 | |
28194 | 33 | 4232 | 1 | 119 | 18099 | 0 | 0 |
(28194,33) | 12 | 2306 | 24 | |
28194 | 34 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28194,34) | 12 | 2306 | 24 | |
28194 | 35 | 3992 | 1 | 119 | 18099 | 0 | 0 |
(28194,35) | 12 | 2306 | 24 | |
28194 | 36 | 3872 | 1 | 117 | 18099 | 0 | 0 |
(28194,36) | 12 | 2306 | 24 | |
28194 | 37 | 3752 | 1 | 119 | 18099 | 0 | 0 |
(28194,37) | 12 | 2306 | 24 | |
28194 | 38 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28194,38) | 12 | 2306 | 24 | |
28194 | 39 | 3512 | 1 | 118 | 18099 | 0 | 0 |
(28194,39) | 12 | 2306 | 24 | |
28194 | 40 | 3392 | 1 | 118 | 18099 | 0 | 0 |
(28194,40) | 12 | 2306 | 24 | |
28194 | 41 | 0 | 3 | 0 | | |
| | | | | |
28194 | 42 | 0 | 3 | 0 | | |
| | | | | |
28194 | 43 | 0 | 3 | 0 | | |
| | | | | |
28194 | 44 | 0 | 3 | 0 | | |
| | | | | |
28194 | 45 | 0 | 3 | 0 | | |
| | | | | |
28194 | 46 | 0 | 3 | 0 | | |
| | | | | |
28194 | 47 | 0 | 3 | 0 | | |
| | | | | |
28194 | 48 | 0 | 3 | 0 | | |
| | | | | |
28194 | 49 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28194,49) | 12 | 2306 | 24 | |
28194 | 50 | 3152 | 1 | 118 | 18099 | 0 | 0 |
(28194,50) | 12 | 2306 | 24 | |
28194 | 51 | 3032 | 1 | 118 | 18099 | 0 | 0 |
(28194,51) | 12 | 2306 | 24 | |
28194 | 52 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28194,52) | 12 | 2306 | 24 | |
28194 | 53 | 2792 | 1 | 118 | 18099 | 0 | 0 |
(28194,53) | 12 | 2306 | 24 | |
28194 | 54 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28194,54) | 12 | 2306 | 24 | |
28194 | 55 | 2552 | 1 | 118 | 18099 | 0 | 0 |
(28194,55) | 12 | 2306 | 24 | |
28194 | 56 | 2432 | 1 | 118 | 18099 | 0 | 0 |
(28194,56) | 12 | 2306 | 24 | |
28194 | 57 | 2312 | 1 | 118 | 18099 | 0 | 0 |
(28194,57) | 12 | 2306 | 24 | |
28194 | 58 | 2192 | 1 | 118 | 18099 | 0 | 0 |
(28194,58) | 12 | 2306 | 24 | |
28194 | 59 | 2072 | 1 | 118 | 18099 | 0 | 0 |
(28194,59) | 12 | 2306 | 24 | |
28194 | 60 | 1952 | 1 | 119 | 18099 | 0 | 0 |
(28194,60) | 12 | 2306 | 24 | |
28194 | 61 | 1832 | 1 | 118 | 18099 | 0 | 0 |
(28194,61) | 12 | 2306 | 24 | |
28194 | 62 | 1712 | 1 | 118 | 18099 | 0 | 0 |
(28194,62) | 12 | 2306 | 24 | |
28194 | 63 | 1592 | 1 | 118 | 18099 | 0 | 0 |
(28194,63) | 12 | 2306 | 24 | |
28194 | 64 | 1472 | 1 | 118 | 18099 | 0 | 0 |
(28194,64) | 12 | 2306 | 24 | |
28194 | 65 | 1352 | 1 | 118 | 18099 | 0 | 0 |
(28194,65) | 12 | 2306 | 24 | |
28194 | 66 | 0 | 0 | 0 | | |
| | | | | |
28194 | 67 | 0 | 0 | 0 | | |
| | | | | |
28194 | 68 | 0 | 0 | 0 | | |
| | | | | |
28194 | 69 | 0 | 0 | 0 | | |
| | | | | |
28194 | 70 | 0 | 0 | 0 | | |
| | | | | |
28194 | 71 | 0 | 0 | 0 | | |
| | | | | |
28194 | 72 | 0 | 0 | 0 | | |
| | | | | |
28194 | 73 | 0 | 0 | 0 | | |
| | | | | |
28203 | 1 | 8072 | 1 | 117 | 18099 | 0 | 0 |
(28203,1) | 12 | 2306 | 24 | |
28203 | 2 | 7952 | 1 | 117 | 18099 | 0 | 0 |
(28203,2) | 12 | 2306 | 24 | |
28203 | 3 | 7832 | 1 | 118 | 18099 | 0 | 0 |
(28203,3) | 12 | 2306 | 24 | |
28203 | 4 | 7712 | 1 | 117 | 224637 | 224637 | 14 |
(28203,4) | 12 | 10642 | 24 | |
28203 | 5 | 7592 | 1 | 118 | 224637 | 224637 | 13 |
(28203,5) | 12 | 10642 | 24 | |
28203 | 6 | 7472 | 1 | 117 | 224637 | 224637 | 12 |
(28203,6) | 12 | 10642 | 24 | |
28203 | 7 | 7352 | 1 | 117 | 224637 | 224637 | 11 |
(28203,7) | 12 | 10642 | 24 | |
28203 | 8 | 7232 | 1 | 117 | 224637 | 224637 | 10 |
(28203,8) | 12 | 10642 | 24 | |
28203 | 9 | 0 | 3 | 0 | | |
| | | | | |
28203 | 10 | 0 | 3 | 0 | | |
| | | | | |
28203 | 11 | 0 | 3 | 0 | | |
| | | | | |
28203 | 12 | 0 | 3 | 0 | | |
| | | | | |
28203 | 13 | 0 | 3 | 0 | | |
| | | | | |
28203 | 14 | 0 | 3 | 0 | | |
| | | | | |
28203 | 15 | 0 | 3 | 0 | | |
| | | | | |
28203 | 16 | 0 | 3 | 0 | | |
| | | | | |
28203 | 17 | 0 | 3 | 0 | | |
| | | | | |
28203 | 18 | 0 | 3 | 0 | | |
| | | | | |
28203 | 19 | 7112 | 1 | 117 | 18099 | 0 | 0 |
(28203,19) | 12 | 2306 | 24 | |
28203 | 20 | 6992 | 1 | 117 | 18099 | 0 | 0 |
(28203,20) | 12 | 2306 | 24 | |
28203 | 21 | 6872 | 1 | 117 | 18099 | 0 | 0 |
(28203,21) | 12 | 2306 | 24 | |
28203 | 22 | 6752 | 1 | 118 | 18099 | 0 | 0 |
(28203,22) | 12 | 2306 | 24 | |
28203 | 23 | 6632 | 1 | 118 | 18099 | 0 | 0 |
(28203,23) | 12 | 2306 | 24 | |
28203 | 24 | 6512 | 1 | 118 | 18099 | 0 | 0 |
(28203,24) | 12 | 2306 | 24 | |
28203 | 25 | 6392 | 1 | 117 | 18099 | 0 | 0 |
(28203,25) | 12 | 2306 | 24 | |
28203 | 26 | 6272 | 1 | 117 | 18099 | 0 | 0 |
(28203,26) | 12 | 2306 | 24 | |
28203 | 27 | 6152 | 1 | 119 | 18099 | 0 | 0 |
(28203,27) | 12 | 2306 | 24 | |
28203 | 28 | 6032 | 1 | 117 | 18099 | 0 | 0 |
(28203,28) | 12 | 2306 | 24 | |
28203 | 29 | 5912 | 1 | 119 | 18099 | 0 | 0 |
(28203,29) | 12 | 2306 | 24 | |
28203 | 30 | 5792 | 1 | 117 | 18099 | 0 | 0 |
(28203,30) | 12 | 2306 | 24 | |
28203 | 31 | 5672 | 1 | 119 | 18099 | 0 | 0 |
(28203,31) | 12 | 2306 | 24 | |
28203 | 32 | 5552 | 1 | 117 | 18099 | 0 | 0 |
(28203,32) | 12 | 2306 | 24 | |
28203 | 33 | 5432 | 1 | 119 | 18099 | 0 | 0 |
(28203,33) | 12 | 2306 | 24 | |
28203 | 34 | 5312 | 1 | 117 | 18099 | 0 | 0 |
(28203,34) | 12 | 2306 | 24 | |
28203 | 35 | 5192 | 1 | 117 | 18099 | 0 | 0 |
(28203,35) | 12 | 2306 | 24 | |
28203 | 36 | 5072 | 1 | 117 | 18099 | 0 | 0 |
(28203,36) | 12 | 2306 | 24 | |
28203 | 37 | 4952 | 1 | 119 | 18099 | 0 | 0 |
(28203,37) | 12 | 2306 | 24 | |
28203 | 38 | 4832 | 1 | 117 | 18099 | 0 | 0 |
(28203,38) | 12 | 2306 | 24 | |
28203 | 39 | 4712 | 1 | 117 | 18099 | 0 | 0 |
(28203,39) | 12 | 2306 | 24 | |
28203 | 40 | 4592 | 1 | 117 | 18099 | 0 | 0 |
(28203,40) | 12 | 2306 | 24 | |
28203 | 41 | 4472 | 1 | 119 | 18099 | 0 | 0 |
(28203,41) | 12 | 2306 | 24 | |
28203 | 42 | 4352 | 1 | 117 | 18099 | 0 | 0 |
(28203,42) | 12 | 2306 | 24 | |
28203 | 43 | 4232 | 1 | 119 | 18099 | 0 | 0 |
(28203,43) | 12 | 2306 | 24 | |
28203 | 44 | 4112 | 1 | 119 | 18099 | 0 | 0 |
(28203,44) | 12 | 2306 | 24 | |
28203 | 45 | 3992 | 1 | 119 | 18099 | 0 | 0 |
(28203,45) | 12 | 2306 | 24 | |
28203 | 46 | 3872 | 1 | 118 | 18099 | 0 | 0 |
(28203,46) | 12 | 2306 | 24 | |
28203 | 47 | 3752 | 1 | 118 | 18099 | 0 | 0 |
(28203,47) | 12 | 2306 | 24 | |
28203 | 48 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28203,48) | 12 | 2306 | 24 | |
28203 | 49 | 3512 | 1 | 118 | 18099 | 0 | 0 |
(28203,49) | 12 | 2306 | 24 | |
28203 | 50 | 3392 | 1 | 118 | 18099 | 0 | 0 |
(28203,50) | 12 | 2306 | 24 | |
28203 | 51 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28203,51) | 12 | 2306 | 24 | |
28203 | 52 | 3152 | 1 | 118 | 18099 | 0 | 0 |
(28203,52) | 12 | 2306 | 24 | |
28203 | 53 | 3032 | 1 | 118 | 18099 | 0 | 0 |
(28203,53) | 12 | 2306 | 24 | |
28203 | 54 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28203,54) | 12 | 2306 | 24 | |
28203 | 55 | 2792 | 1 | 118 | 18099 | 0 | 0 |
(28203,55) | 12 | 2306 | 24 | |
28203 | 56 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28203,56) | 12 | 2306 | 24 | |
28203 | 57 | 2552 | 1 | 118 | 18099 | 0 | 0 |
(28203,57) | 12 | 2306 | 24 | |
28203 | 58 | 2432 | 1 | 117 | 18099 | 0 | 0 |
(28203,58) | 12 | 2306 | 24 | |
28203 | 59 | 2312 | 1 | 117 | 18099 | 0 | 0 |
(28203,59) | 12 | 2306 | 24 | |
28203 | 60 | 2192 | 1 | 118 | 18099 | 0 | 0 |
(28203,60) | 12 | 2306 | 24 | |
28203 | 61 | 2072 | 1 | 117 | 18099 | 0 | 0 |
(28203,61) | 12 | 2306 | 24 | |
28203 | 62 | 1952 | 1 | 119 | 18099 | 0 | 0 |
(28203,62) | 12 | 2306 | 24 | |
28203 | 63 | 1832 | 1 | 119 | 18099 | 0 | 0 |
(28203,63) | 12 | 2306 | 24 | |
28203 | 64 | 1712 | 1 | 119 | 18099 | 0 | 0 |
(28203,64) | 12 | 2306 | 24 | |
28203 | 65 | 1592 | 1 | 117 | 18099 | 0 | 0 |
(28203,65) | 12 | 2306 | 24 | |
28204 | 1 | 8072 | 1 | 117 | 18099 | 0 | 0 |
(28204,1) | 12 | 2306 | 24 | |
28204 | 2 | 7952 | 1 | 117 | 18099 | 0 | 0 |
(28204,2) | 12 | 2306 | 24 | |
28204 | 3 | 7832 | 1 | 117 | 18099 | 0 | 0 |
(28204,3) | 12 | 2306 | 24 | |
28204 | 4 | 7712 | 1 | 119 | 18099 | 0 | 0 |
(28204,4) | 12 | 2306 | 24 | |
28204 | 5 | 7592 | 1 | 119 | 18099 | 0 | 0 |
(28204,5) | 12 | 2306 | 24 | |
28204 | 6 | 7472 | 1 | 117 | 18099 | 0 | 0 |
(28204,6) | 12 | 2306 | 24 | |
28204 | 7 | 7352 | 1 | 118 | 18099 | 0 | 0 |
(28204,7) | 12 | 2306 | 24 | |
28204 | 8 | 7232 | 1 | 118 | 18099 | 0 | 0 |
(28204,8) | 12 | 2306 | 24 | |
28204 | 9 | 7112 | 1 | 119 | 18099 | 0 | 0 |
(28204,9) | 12 | 2306 | 24 | |
28204 | 10 | 6992 | 1 | 117 | 18099 | 0 | 0 |
(28204,10) | 12 | 2306 | 24 | |
28204 | 11 | 6872 | 1 | 117 | 18099 | 0 | 0 |
(28204,11) | 12 | 2306 | 24 | |
28204 | 12 | 6752 | 1 | 117 | 18099 | 0 | 0 |
(28204,12) | 12 | 2306 | 24 | |
28204 | 13 | 6632 | 1 | 117 | 18099 | 0 | 0 |
(28204,13) | 12 | 2306 | 24 | |
28204 | 14 | 6512 | 1 | 117 | 18099 | 0 | 0 |
(28204,14) | 12 | 2306 | 24 | |
28204 | 15 | 6392 | 1 | 119 | 18099 | 0 | 0 |
(28204,15) | 12 | 2306 | 24 | |
28204 | 16 | 6272 | 1 | 119 | 18099 | 0 | 0 |
(28204,16) | 12 | 2306 | 24 | |
28204 | 17 | 6152 | 1 | 119 | 18099 | 0 | 0 |
(28204,17) | 12 | 2306 | 24 | |
28204 | 18 | 6032 | 1 | 117 | 18099 | 0 | 0 |
(28204,18) | 12 | 2306 | 24 | |
28204 | 19 | 5912 | 1 | 119 | 18099 | 0 | 0 |
(28204,19) | 12 | 2306 | 24 | |
28204 | 20 | 5792 | 1 | 117 | 18099 | 0 | 0 |
(28204,20) | 12 | 2306 | 24 | |
28204 | 21 | 5672 | 1 | 117 | 18099 | 0 | 0 |
(28204,21) | 12 | 2306 | 24 | |
28204 | 22 | 5552 | 1 | 117 | 18099 | 0 | 0 |
(28204,22) | 12 | 2306 | 24 | |
28204 | 23 | 5432 | 1 | 117 | 18099 | 0 | 0 |
(28204,23) | 12 | 2306 | 24 | |
28204 | 24 | 5312 | 1 | 119 | 18099 | 0 | 0 |
(28204,24) | 12 | 2306 | 24 | |
28204 | 25 | 5192 | 1 | 117 | 18099 | 0 | 0 |
(28204,25) | 12 | 2306 | 24 | |
28204 | 26 | 5072 | 1 | 119 | 224663 | 224663 | 8 |
(28204,26) | 12 | 10642 | 24 | |
28204 | 27 | 4952 | 1 | 119 | 224663 | 224663 | 7 |
(28204,27) | 12 | 10642 | 24 | |
28204 | 28 | 4832 | 1 | 119 | 224663 | 224663 | 6 |
(28204,28) | 12 | 10642 | 24 | |
28204 | 29 | 4712 | 1 | 118 | 224663 | 224663 | 5 |
(28204,29) | 12 | 10642 | 24 | |
28204 | 30 | 4592 | 1 | 117 | 224663 | 224663 | 4 |
(28204,30) | 12 | 10642 | 24 | |
28204 | 31 | 4472 | 1 | 119 | 224663 | 224663 | 3 |
(28204,31) | 12 | 10642 | 24 | |
28204 | 32 | 4352 | 1 | 119 | 224663 | 208645 | 2 |
(29998,54) | 8204 | 12610 | 24 | |
28204 | 33 | 4232 | 1 | 119 | 224663 | 208645 | 1 |
(29998,53) | 8204 | 12610 | 24 | |
28204 | 34 | 4112 | 1 | 119 | 224663 | 208645 | 0 |
(29998,52) | 8204 | 12610 | 24 | |
28204 | 35 | 3992 | 1 | 117 | 18099 | 0 | 0 |
(28204,35) | 12 | 2306 | 24 | |
28204 | 36 | 3872 | 1 | 117 | 18099 | 0 | 0 |
(28204,36) | 12 | 2306 | 24 | |
28204 | 37 | 3752 | 1 | 117 | 18099 | 0 | 0 |
(28204,37) | 12 | 2306 | 24 | |
28204 | 38 | 3632 | 1 | 117 | 18099 | 0 | 0 |
(28204,38) | 12 | 2306 | 24 | |
28204 | 39 | 3512 | 1 | 117 | 18099 | 0 | 0 |
(28204,39) | 12 | 2306 | 24 | |
28204 | 40 | 3392 | 1 | 117 | 18099 | 0 | 0 |
(28204,40) | 12 | 2306 | 24 | |
28204 | 41 | 3272 | 1 | 117 | 18099 | 0 | 0 |
(28204,41) | 12 | 2306 | 24 | |
28204 | 42 | 3152 | 1 | 117 | 18099 | 0 | 0 |
(28204,42) | 12 | 2306 | 24 | |
28204 | 43 | 3032 | 1 | 117 | 18099 | 0 | 0 |
(28204,43) | 12 | 2306 | 24 | |
28204 | 44 | 2912 | 1 | 117 | 18099 | 0 | 0 |
(28204,44) | 12 | 2306 | 24 | |
28204 | 45 | 2792 | 1 | 117 | 18099 | 0 | 0 |
(28204,45) | 12 | 2306 | 24 | |
28204 | 46 | 2672 | 1 | 117 | 18099 | 0 | 0 |
(28204,46) | 12 | 2306 | 24 | |
28204 | 47 | 2552 | 1 | 117 | 18099 | 0 | 0 |
(28204,47) | 12 | 2306 | 24 | |
28204 | 48 | 2432 | 1 | 117 | 18099 | 0 | 0 |
(28204,48) | 12 | 2306 | 24 | |
28204 | 49 | 2312 | 1 | 117 | 18099 | 0 | 0 |
(28204,49) | 12 | 2306 | 24 | |
28204 | 50 | 2192 | 1 | 117 | 18099 | 0 | 0 |
(28204,50) | 12 | 2306 | 24 | |
28204 | 51 | 2072 | 1 | 117 | 18099 | 0 | 0 |
(28204,51) | 12 | 2306 | 24 | |
28204 | 52 | 1952 | 1 | 117 | 18099 | 0 | 0 |
(28204,52) | 12 | 2306 | 24 | |
28204 | 53 | 1832 | 1 | 119 | 18099 | 0 | 0 |
(28204,53) | 12 | 2306 | 24 | |
28204 | 54 | 1712 | 1 | 119 | 18099 | 0 | 0 |
(28204,54) | 12 | 2306 | 24 | |
28204 | 55 | 1592 | 1 | 117 | 18099 | 0 | 0 |
(28204,55) | 12 | 2306 | 24 | |
28204 | 56 | 1472 | 1 | 117 | 18099 | 0 | 0 |
(28204,56) | 12 | 2306 | 24 | |
28204 | 57 | 1352 | 1 | 117 | 18099 | 0 | 0 |
(28204,57) | 12 | 2306 | 24 | |
28204 | 58 | 1232 | 1 | 117 | 18099 | 0 | 0 |
(28204,58) | 12 | 2306 | 24 | |
28204 | 59 | 1112 | 1 | 117 | 18099 | 0 | 0 |
(28204,59) | 12 | 2306 | 24 | |
28204 | 60 | 992 | 1 | 117 | 18099 | 0 | 0 |
(28204,60) | 12 | 2306 | 24 | |
28204 | 61 | 872 | 1 | 117 | 18099 | 0 | 0 |
(28204,61) | 12 | 2306 | 24 | |
28204 | 62 | 752 | 1 | 117 | 18099 | 0 | 0 |
(28204,62) | 12 | 2306 | 24 | |
28204 | 63 | 632 | 1 | 117 | 18099 | 0 | 0 |
(28204,63) | 12 | 2306 | 24 | |
28204 | 64 | 512 | 1 | 117 | 18099 | 0 | 0 |
(28204,64) | 12 | 2306 | 24 | |
28204 | 65 | 392 | 1 | 117 | 18099 | 0 | 0 |
(28204,65) | 12 | 2306 | 24 | |
28281 | 1 | 8072 | 1 | 117 | 224832 | 224832 | 105 |
(28281,1) | 12 | 10642 | 24 | |
28281 | 2 | 7952 | 1 | 117 | 224832 | 224832 | 104 |
(28281,2) | 12 | 10642 | 24 | |
28281 | 3 | 7832 | 1 | 119 | 224832 | 224832 | 103 |
(28281,3) | 12 | 10642 | 24 | |
28281 | 4 | 7712 | 1 | 117 | 224832 | 224832 | 102 |
(28281,4) | 12 | 10642 | 24 | |
28281 | 5 | 7592 | 1 | 117 | 224832 | 224832 | 101 |
(28281,5) | 12 | 10642 | 24 | |
28281 | 6 | 7472 | 1 | 117 | 224832 | 224832 | 100 |
(28281,6) | 12 | 10642 | 24 | |
28281 | 7 | 7352 | 1 | 117 | 224832 | 224832 | 99 |
(28281,7) | 12 | 10642 | 24 | |
28281 | 8 | 7232 | 1 | 117 | 224832 | 224832 | 98 |
(28281,8) | 12 | 10642 | 24 | |
28281 | 9 | 7112 | 1 | 117 | 224832 | 224832 | 97 |
(28281,9) | 12 | 10642 | 24 | |
28281 | 10 | 6992 | 1 | 119 | 224832 | 224832 | 96 |
(28281,10) | 12 | 10642 | 24 | |
28281 | 11 | 6872 | 1 | 117 | 224832 | 224832 | 95 |
(28281,11) | 12 | 10642 | 24 | |
28281 | 12 | 6752 | 1 | 117 | 224832 | 224832 | 94 |
(28281,12) | 12 | 10642 | 24 | |
28281 | 13 | 6632 | 1 | 117 | 224832 | 224832 | 93 |
(28281,13) | 12 | 10642 | 24 | |
28281 | 14 | 6512 | 1 | 117 | 224832 | 224832 | 92 |
(28281,14) | 12 | 10642 | 24 | |
28281 | 15 | 6392 | 1 | 119 | 224832 | 224832 | 91 |
(28281,15) | 12 | 10642 | 24 | |
28281 | 16 | 6272 | 1 | 117 | 224832 | 224832 | 90 |
(28281,16) | 12 | 10642 | 24 | |
28281 | 17 | 6152 | 1 | 117 | 224832 | 224832 | 89 |
(28281,17) | 12 | 10642 | 24 | |
28281 | 18 | 6032 | 1 | 117 | 224832 | 224832 | 88 |
(28281,18) | 12 | 10642 | 24 | |
28281 | 19 | 5912 | 1 | 119 | 224832 | 224832 | 87 |
(28281,19) | 12 | 10642 | 24 | |
28281 | 20 | 5792 | 1 | 117 | 224832 | 224832 | 86 |
(28281,20) | 12 | 10642 | 24 | |
28281 | 21 | 5672 | 1 | 117 | 224832 | 224832 | 85 |
(28281,21) | 12 | 10642 | 24 | |
28281 | 22 | 5552 | 1 | 117 | 224832 | 224832 | 84 |
(28281,22) | 12 | 10642 | 24 | |
28281 | 23 | 5432 | 1 | 117 | 224832 | 224832 | 83 |
(28281,23) | 12 | 10642 | 24 | |
28281 | 24 | 5312 | 1 | 117 | 224832 | 224832 | 82 |
(28281,24) | 12 | 10642 | 24 | |
28281 | 25 | 5192 | 1 | 117 | 224832 | 224832 | 81 |
(28281,25) | 12 | 10642 | 24 | |
28281 | 26 | 5072 | 1 | 117 | 224832 | 224832 | 80 |
(28281,26) | 12 | 10642 | 24 | |
28281 | 27 | 4952 | 1 | 117 | 224832 | 224832 | 79 |
(28281,27) | 12 | 10642 | 24 | |
28281 | 28 | 4832 | 1 | 117 | 224832 | 224832 | 78 |
(28281,28) | 12 | 10642 | 24 | |
28281 | 29 | 4712 | 1 | 117 | 224832 | 224832 | 77 |
(28281,29) | 12 | 10642 | 24 | |
28281 | 30 | 4592 | 1 | 117 | 224832 | 208723 | 125 |
(30008,58) | 8204 | 12610 | 24 | |
28281 | 31 | 4472 | 1 | 117 | 224832 | 208723 | 124 |
(30008,57) | 8204 | 12610 | 24 | |
28281 | 32 | 4352 | 1 | 117 | 224832 | 208723 | 123 |
(30008,56) | 8204 | 12610 | 24 | |
28281 | 33 | 4232 | 1 | 117 | 224832 | 208723 | 122 |
(30008,55) | 8204 | 12610 | 24 | |
28281 | 34 | 4112 | 1 | 117 | 224832 | 208723 | 121 |
(30008,54) | 8204 | 12610 | 24 | |
28281 | 35 | 3992 | 1 | 119 | 224832 | 208723 | 120 |
(30008,53) | 8204 | 12610 | 24 | |
28281 | 36 | 3872 | 1 | 117 | 224832 | 208723 | 119 |
(30008,52) | 8204 | 12610 | 24 | |
28281 | 37 | 3752 | 1 | 117 | 224832 | 208723 | 118 |
(30008,51) | 8204 | 12610 | 24 | |
28281 | 38 | 3632 | 1 | 118 | 224832 | 208723 | 117 |
(30008,50) | 8204 | 12610 | 24 | |
28281 | 39 | 3512 | 1 | 117 | 224832 | 208723 | 116 |
(30008,49) | 8204 | 12610 | 24 | |
28281 | 40 | 3392 | 1 | 117 | 224832 | 208723 | 115 |
(30008,48) | 8204 | 12610 | 24 | |
28281 | 41 | 3272 | 1 | 117 | 224832 | 208723 | 114 |
(30008,47) | 8204 | 12610 | 24 | |
28281 | 42 | 3152 | 1 | 117 | 224832 | 208723 | 113 |
(30008,46) | 8204 | 12610 | 24 | |
28281 | 43 | 3032 | 1 | 117 | 224832 | 208723 | 112 |
(30008,45) | 8204 | 12610 | 24 | |
28281 | 44 | 2912 | 1 | 117 | 224832 | 208723 | 111 |
(30008,44) | 8204 | 12610 | 24 | |
28281 | 45 | 2792 | 1 | 117 | 224832 | 208723 | 110 |
(30008,43) | 8204 | 12610 | 24 | |
28281 | 46 | 2672 | 1 | 117 | 224832 | 208723 | 109 |
(30008,42) | 8204 | 12610 | 24 | |
28281 | 47 | 2552 | 1 | 117 | 224832 | 208723 | 108 |
(30008,41) | 8204 | 12610 | 24 | |
28281 | 48 | 2432 | 1 | 117 | 224832 | 208723 | 107 |
(30008,40) | 8204 | 12610 | 24 | |
28281 | 49 | 2312 | 1 | 117 | 224832 | 208723 | 106 |
(30008,39) | 8204 | 12610 | 24 | |
28281 | 50 | 2192 | 1 | 117 | 224832 | 208723 | 105 |
(30008,38) | 8204 | 12610 | 24 | |
28281 | 51 | 2072 | 1 | 117 | 224832 | 208723 | 104 |
(30008,37) | 8204 | 12610 | 24 | |
28281 | 52 | 1952 | 1 | 117 | 224832 | 208723 | 103 |
(30008,36) | 8204 | 12610 | 24 | |
28281 | 53 | 1832 | 1 | 117 | 224832 | 208723 | 102 |
(30008,35) | 8204 | 12610 | 24 | |
28281 | 54 | 1712 | 1 | 117 | 224832 | 208723 | 101 |
(30008,34) | 8204 | 12610 | 24 | |
28281 | 55 | 1592 | 1 | 119 | 224832 | 208723 | 100 |
(30008,33) | 8204 | 12610 | 24 | |
28281 | 56 | 1472 | 1 | 117 | 224832 | 208723 | 99 |
(30008,32) | 8204 | 12610 | 24 | |
28281 | 57 | 1352 | 1 | 117 | 224832 | 208723 | 98 |
(30008,31) | 8204 | 12610 | 24 | |
28281 | 58 | 1232 | 1 | 117 | 224832 | 208723 | 97 |
(30008,30) | 8204 | 12610 | 24 | |
28281 | 59 | 1112 | 1 | 119 | 224832 | 208723 | 96 |
(30008,29) | 8204 | 12610 | 24 | |
28281 | 60 | 992 | 1 | 117 | 224832 | 208723 | 95 |
(30008,28) | 8204 | 12610 | 24 | |
28281 | 61 | 872 | 1 | 117 | 224832 | 208723 | 94 |
(30008,27) | 8204 | 12610 | 24 | |
28281 | 62 | 752 | 1 | 117 | 224832 | 208723 | 93 |
(30008,26) | 8204 | 12610 | 24 | |
28281 | 63 | 632 | 1 | 117 | 224832 | 208723 | 92 |
(30008,25) | 8204 | 12610 | 24 | |
28281 | 64 | 512 | 1 | 117 | 224832 | 208723 | 91 |
(30008,24) | 8204 | 12610 | 24 | |
28281 | 65 | 392 | 1 | 117 | 224832 | 208723 | 90 |
(30008,23) | 8204 | 12610 | 24 | |
28390 | 1 | 8072 | 1 | 118 | 18099 | 0 | 0 |
(28390,1) | 12 | 2306 | 24 | |
28390 | 2 | 7952 | 1 | 118 | 18099 | 0 | 0 |
(28390,2) | 12 | 2306 | 24 | |
28390 | 3 | 7832 | 1 | 118 | 18099 | 0 | 0 |
(28390,3) | 12 | 2306 | 24 | |
28390 | 4 | 7712 | 1 | 118 | 18099 | 0 | 0 |
(28390,4) | 12 | 2306 | 24 | |
28390 | 5 | 7592 | 1 | 118 | 18099 | 0 | 0 |
(28390,5) | 12 | 2306 | 24 | |
28390 | 6 | 7472 | 1 | 118 | 18099 | 0 | 0 |
(28390,6) | 12 | 2306 | 24 | |
28390 | 7 | 7352 | 1 | 118 | 18099 | 0 | 0 |
(28390,7) | 12 | 2306 | 24 | |
28390 | 8 | 7232 | 1 | 118 | 18099 | 0 | 0 |
(28390,8) | 12 | 2306 | 24 | |
28390 | 9 | 7112 | 1 | 118 | 18099 | 0 | 0 |
(28390,9) | 12 | 2306 | 24 | |
28390 | 10 | 6992 | 1 | 118 | 18099 | 0 | 0 |
(28390,10) | 12 | 2306 | 24 | |
28390 | 11 | 6872 | 1 | 119 | 18099 | 0 | 0 |
(28390,11) | 12 | 2306 | 24 | |
28390 | 12 | 6752 | 1 | 118 | 225505 | 225505 | 10 |
(28390,12) | 12 | 10642 | 24 | |
28390 | 13 | 6632 | 1 | 118 | 225505 | 225505 | 9 |
(28390,13) | 12 | 10642 | 24 | |
28390 | 14 | 6512 | 1 | 118 | 225505 | 225505 | 8 |
(28390,14) | 12 | 10642 | 24 | |
28390 | 15 | 6392 | 1 | 118 | 225505 | 225505 | 7 |
(28390,15) | 12 | 10642 | 24 | |
28390 | 16 | 6272 | 1 | 118 | 225505 | 225505 | 6 |
(28390,16) | 12 | 10642 | 24 | |
28390 | 17 | 6152 | 1 | 118 | 225505 | 225505 | 5 |
(28390,17) | 12 | 10642 | 24 | |
28390 | 18 | 6032 | 1 | 119 | 225505 | 225505 | 4 |
(28390,18) | 12 | 10642 | 24 | |
28390 | 19 | 5912 | 1 | 118 | 225505 | 208929 | 3 |
(29994,36) | 8204 | 12610 | 24 | |
28390 | 20 | 5792 | 1 | 118 | 225505 | 208929 | 2 |
(29994,35) | 8204 | 12610 | 24 | |
28390 | 21 | 5672 | 1 | 118 | 225505 | 208929 | 1 |
(29994,34) | 8204 | 12610 | 24 | |
28390 | 22 | 5552 | 1 | 118 | 225505 | 208929 | 0 |
(29994,33) | 8204 | 12610 | 24 | |
28390 | 23 | 5432 | 1 | 118 | 18099 | 0 | 0 |
(28390,23) | 12 | 2306 | 24 | |
28390 | 24 | 5312 | 1 | 118 | 18099 | 0 | 0 |
(28390,24) | 12 | 2306 | 24 | |
28390 | 25 | 5192 | 1 | 118 | 18099 | 0 | 0 |
(28390,25) | 12 | 2306 | 24 | |
28390 | 26 | 5072 | 1 | 118 | 18099 | 0 | 0 |
(28390,26) | 12 | 2306 | 24 | |
28390 | 27 | 4952 | 1 | 118 | 18099 | 0 | 0 |
(28390,27) | 12 | 2306 | 24 | |
28390 | 28 | 4832 | 1 | 118 | 18099 | 0 | 0 |
(28390,28) | 12 | 2306 | 24 | |
28390 | 29 | 4712 | 1 | 118 | 18099 | 0 | 0 |
(28390,29) | 12 | 2306 | 24 | |
28390 | 30 | 4592 | 1 | 118 | 18099 | 0 | 0 |
(28390,30) | 12 | 2306 | 24 | |
28390 | 31 | 4472 | 1 | 118 | 18099 | 0 | 0 |
(28390,31) | 12 | 2306 | 24 | |
28390 | 32 | 4352 | 1 | 118 | 18099 | 0 | 0 |
(28390,32) | 12 | 2306 | 24 | |
28390 | 33 | 4232 | 1 | 117 | 18099 | 0 | 0 |
(28390,33) | 12 | 2306 | 24 | |
28390 | 34 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28390,34) | 12 | 2306 | 24 | |
28390 | 35 | 3992 | 1 | 118 | 18099 | 0 | 0 |
(28390,35) | 12 | 2306 | 24 | |
28390 | 36 | 3872 | 1 | 118 | 18099 | 0 | 0 |
(28390,36) | 12 | 2306 | 24 | |
28390 | 37 | 3752 | 1 | 119 | 18099 | 0 | 0 |
(28390,37) | 12 | 2306 | 24 | |
28390 | 38 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28390,38) | 12 | 2306 | 24 | |
28390 | 39 | 3512 | 1 | 117 | 18099 | 0 | 0 |
(28390,39) | 12 | 2306 | 24 | |
28390 | 40 | 3392 | 1 | 118 | 18099 | 0 | 0 |
(28390,40) | 12 | 2306 | 24 | |
28390 | 41 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28390,41) | 12 | 2306 | 24 | |
28390 | 42 | 3152 | 1 | 117 | 18099 | 0 | 0 |
(28390,42) | 12 | 2306 | 24 | |
28390 | 43 | 3032 | 1 | 117 | 18099 | 0 | 0 |
(28390,43) | 12 | 2306 | 24 | |
28390 | 44 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28390,44) | 12 | 2306 | 24 | |
28390 | 45 | 2792 | 1 | 118 | 18099 | 0 | 0 |
(28390,45) | 12 | 2306 | 24 | |
28390 | 46 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28390,46) | 12 | 2306 | 24 | |
28390 | 47 | 2552 | 1 | 119 | 18099 | 0 | 0 |
(28390,47) | 12 | 2306 | 24 | |
28390 | 48 | 2432 | 1 | 117 | 18099 | 0 | 0 |
(28390,48) | 12 | 2306 | 24 | |
28390 | 49 | 2312 | 1 | 118 | 18099 | 0 | 0 |
(28390,49) | 12 | 2306 | 24 | |
28390 | 50 | 2192 | 1 | 117 | 18099 | 0 | 0 |
(28390,50) | 12 | 2306 | 24 | |
28390 | 51 | 2072 | 1 | 118 | 18099 | 0 | 0 |
(28390,51) | 12 | 2306 | 24 | |
28390 | 52 | 1952 | 1 | 118 | 18099 | 0 | 0 |
(28390,52) | 12 | 2306 | 24 | |
28390 | 53 | 1832 | 1 | 117 | 18099 | 0 | 0 |
(28390,53) | 12 | 2306 | 24 | |
28390 | 54 | 1712 | 1 | 118 | 18099 | 0 | 0 |
(28390,54) | 12 | 2306 | 24 | |
28390 | 55 | 1592 | 1 | 118 | 18099 | 0 | 0 |
(28390,55) | 12 | 2306 | 24 | |
28390 | 56 | 1472 | 1 | 118 | 18099 | 0 | 0 |
(28390,56) | 12 | 2306 | 24 | |
28390 | 57 | 1352 | 1 | 118 | 18099 | 0 | 0 |
(28390,57) | 12 | 2306 | 24 | |
28390 | 58 | 1232 | 1 | 119 | 18099 | 0 | 0 |
(28390,58) | 12 | 2306 | 24 | |
28390 | 59 | 1112 | 1 | 117 | 18099 | 0 | 0 |
(28390,59) | 12 | 2306 | 24 | |
28390 | 60 | 992 | 1 | 118 | 18099 | 0 | 0 |
(28390,60) | 12 | 2306 | 24 | |
28390 | 61 | 872 | 1 | 118 | 18099 | 0 | 0 |
(28390,61) | 12 | 2306 | 24 | |
28390 | 62 | 752 | 1 | 118 | 18099 | 0 | 0 |
(28390,62) | 12 | 2306 | 24 | |
28390 | 63 | 632 | 1 | 118 | 18099 | 0 | 0 |
(28390,63) | 12 | 2306 | 24 | |
28390 | 64 | 512 | 1 | 117 | 18099 | 0 | 0 |
(28390,64) | 12 | 2306 | 24 | |
28390 | 65 | 392 | 1 | 118 | 18099 | 0 | 0 |
(28390,65) | 12 | 2306 | 24 | |
28392 | 1 | 0 | 3 | 0 | | |
| | | | | |
28392 | 2 | 0 | 3 | 0 | | |
| | | | | |
28392 | 3 | 0 | 3 | 0 | | |
| | | | | |
28392 | 4 | 8072 | 1 | 118 | 18099 | 0 | 0 |
(28392,4) | 12 | 2306 | 24 | |
28392 | 5 | 0 | 3 | 0 | | |
| | | | | |
28392 | 6 | 7952 | 1 | 118 | 18099 | 0 | 0 |
(28392,6) | 12 | 2306 | 24 | |
28392 | 7 | 0 | 3 | 0 | | |
| | | | | |
28392 | 8 | 7832 | 1 | 119 | 18099 | 0 | 0 |
(28392,8) | 12 | 2306 | 24 | |
28392 | 9 | 7712 | 1 | 119 | 18099 | 0 | 0 |
(28392,9) | 12 | 2306 | 24 | |
28392 | 10 | 7592 | 1 | 117 | 18099 | 0 | 0 |
(28392,10) | 12 | 2306 | 24 | |
28392 | 11 | 7472 | 1 | 117 | 18099 | 0 | 0 |
(28392,11) | 12 | 2306 | 24 | |
28392 | 12 | 7352 | 1 | 117 | 18099 | 0 | 0 |
(28392,12) | 12 | 2306 | 24 | |
28392 | 13 | 7232 | 1 | 117 | 18099 | 0 | 0 |
(28392,13) | 12 | 2306 | 24 | |
28392 | 14 | 7112 | 1 | 119 | 18099 | 0 | 0 |
(28392,14) | 12 | 2306 | 24 | |
28392 | 15 | 6992 | 1 | 118 | 18099 | 0 | 0 |
(28392,15) | 12 | 2306 | 24 | |
28392 | 16 | 6872 | 1 | 118 | 18099 | 0 | 0 |
(28392,16) | 12 | 2306 | 24 | |
28392 | 17 | 6752 | 1 | 117 | 18099 | 0 | 0 |
(28392,17) | 12 | 2306 | 24 | |
28392 | 18 | 6632 | 1 | 117 | 18099 | 0 | 0 |
(28392,18) | 12 | 2306 | 24 | |
28392 | 19 | 6512 | 1 | 117 | 18099 | 0 | 0 |
(28392,19) | 12 | 2306 | 24 | |
28392 | 20 | 6392 | 1 | 117 | 18099 | 0 | 0 |
(28392,20) | 12 | 2306 | 24 | |
28392 | 21 | 6272 | 1 | 117 | 18099 | 0 | 0 |
(28392,21) | 12 | 2306 | 24 | |
28392 | 22 | 6152 | 1 | 117 | 18099 | 0 | 0 |
(28392,22) | 12 | 2306 | 24 | |
28392 | 23 | 6032 | 1 | 117 | 18099 | 0 | 0 |
(28392,23) | 12 | 2306 | 24 | |
28392 | 24 | 5912 | 1 | 117 | 18099 | 0 | 0 |
(28392,24) | 12 | 2306 | 24 | |
28392 | 25 | 66 | 2 | 0 | | |
| | | | | |
28392 | 26 | 5792 | 1 | 117 | 18099 | 0 | 0 |
(28392,26) | 12 | 2306 | 24 | |
28392 | 27 | 5672 | 1 | 117 | 18099 | 0 | 0 |
(28392,27) | 12 | 2306 | 24 | |
28392 | 28 | 5552 | 1 | 117 | 18099 | 0 | 0 |
(28392,28) | 12 | 2306 | 24 | |
28392 | 29 | 5432 | 1 | 117 | 18099 | 0 | 0 |
(28392,29) | 12 | 2306 | 24 | |
28392 | 30 | 5312 | 1 | 117 | 18099 | 0 | 0 |
(28392,30) | 12 | 2306 | 24 | |
28392 | 31 | 5192 | 1 | 119 | 18099 | 0 | 0 |
(28392,31) | 12 | 2306 | 24 | |
28392 | 32 | 5072 | 1 | 117 | 18099 | 0 | 0 |
(28392,32) | 12 | 2306 | 24 | |
28392 | 33 | 4952 | 1 | 119 | 18099 | 0 | 0 |
(28392,33) | 12 | 2306 | 24 | |
28392 | 34 | 4832 | 1 | 117 | 18099 | 0 | 0 |
(28392,34) | 12 | 2306 | 24 | |
28392 | 35 | 4712 | 1 | 117 | 18099 | 0 | 0 |
(28392,35) | 12 | 2306 | 24 | |
28392 | 36 | 4592 | 1 | 117 | 18099 | 0 | 0 |
(28392,36) | 12 | 2306 | 24 | |
28392 | 37 | 4472 | 1 | 117 | 18099 | 0 | 0 |
(28392,37) | 12 | 2306 | 24 | |
28392 | 38 | 4352 | 1 | 118 | 18099 | 0 | 0 |
(28392,38) | 12 | 2306 | 24 | |
28392 | 39 | 4232 | 1 | 117 | 18099 | 0 | 0 |
(28392,39) | 12 | 2306 | 24 | |
28392 | 40 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28392,40) | 12 | 2306 | 24 | |
28392 | 41 | 3992 | 1 | 119 | 18099 | 0 | 0 |
(28392,41) | 12 | 2306 | 24 | |
28392 | 42 | 3872 | 1 | 119 | 225523 | 208938 | 2 |
(28392,69) | 24588 | 14658 | 24 | |
28392 | 43 | 3752 | 1 | 118 | 225523 | 208938 | 1 |
(28392,68) | 24588 | 14658 | 24 | |
28392 | 44 | 3632 | 1 | 119 | 225523 | 208938 | 0 |
(28392,67) | 24588 | 14658 | 24 | |
28392 | 45 | 3512 | 1 | 118 | 18099 | 0 | 0 |
(28392,45) | 12 | 2306 | 24 | |
28392 | 46 | 3392 | 1 | 117 | 18099 | 0 | 0 |
(28392,46) | 12 | 2306 | 24 | |
28392 | 47 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28392,47) | 12 | 2306 | 24 | |
28392 | 48 | 3152 | 1 | 117 | 18099 | 0 | 0 |
(28392,48) | 12 | 2306 | 24 | |
28392 | 49 | 3032 | 1 | 117 | 18099 | 0 | 0 |
(28392,49) | 12 | 2306 | 24 | |
28392 | 50 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28392,50) | 12 | 2306 | 24 | |
28392 | 51 | 2792 | 1 | 117 | 18099 | 0 | 0 |
(28392,51) | 12 | 2306 | 24 | |
28392 | 52 | 2672 | 1 | 119 | 18099 | 0 | 0 |
(28392,52) | 12 | 2306 | 24 | |
28392 | 53 | 2552 | 1 | 117 | 18099 | 0 | 0 |
(28392,53) | 12 | 2306 | 24 | |
28392 | 54 | 2432 | 1 | 117 | 18099 | 0 | 0 |
(28392,54) | 12 | 2306 | 24 | |
28392 | 55 | 2312 | 1 | 117 | 18099 | 0 | 0 |
(28392,55) | 12 | 2306 | 24 | |
28392 | 56 | 2192 | 1 | 117 | 18099 | 0 | 0 |
(28392,56) | 12 | 2306 | 24 | |
28392 | 57 | 2072 | 1 | 117 | 18099 | 0 | 0 |
(28392,57) | 12 | 2306 | 24 | |
28392 | 58 | 1952 | 1 | 119 | 18099 | 0 | 0 |
(28392,58) | 12 | 2306 | 24 | |
28392 | 59 | 1832 | 1 | 117 | 18099 | 0 | 0 |
(28392,59) | 12 | 2306 | 24 | |
28392 | 60 | 1712 | 1 | 118 | 18099 | 0 | 0 |
(28392,60) | 12 | 2306 | 24 | |
28392 | 61 | 1592 | 1 | 117 | 18099 | 0 | 0 |
(28392,61) | 12 | 2306 | 24 | |
28392 | 62 | 1472 | 1 | 118 | 18099 | 0 | 0 |
(28392,62) | 12 | 2306 | 24 | |
28392 | 63 | 1352 | 1 | 117 | 18099 | 0 | 0 |
(28392,63) | 12 | 2306 | 24 | |
28392 | 64 | 1232 | 1 | 117 | 18099 | 0 | 0 |
(28392,64) | 12 | 2306 | 24 | |
28392 | 65 | 1112 | 1 | 117 | 18099 | 0 | 0 |
(28392,65) | 12 | 2306 | 24 | |
28392 | 66 | 992 | 1 | 118 | 570583 | 570583 | 0 |
(28392,66) | 32780 | 10642 | 24 | |
28392 | 67 | 0 | 0 | 0 | | |
| | | | | |
28392 | 68 | 0 | 0 | 0 | | |
| | | | | |
28392 | 69 | 0 | 0 | 0 | | |
| | | | | |
28410 | 1 | 8072 | 1 | 118 | 18099 | 0 | 0 |
(28410,1) | 12 | 2306 | 24 | |
28410 | 2 | 7952 | 1 | 117 | 18099 | 0 | 0 |
(28410,2) | 12 | 2306 | 24 | |
28410 | 3 | 7832 | 1 | 117 | 18099 | 0 | 0 |
(28410,3) | 12 | 2306 | 24 | |
28410 | 4 | 7712 | 1 | 117 | 18099 | 0 | 0 |
(28410,4) | 12 | 2306 | 24 | |
28410 | 5 | 7592 | 1 | 117 | 18099 | 0 | 0 |
(28410,5) | 12 | 2306 | 24 | |
28410 | 6 | 7472 | 1 | 119 | 18099 | 0 | 0 |
(28410,6) | 12 | 2306 | 24 | |
28410 | 7 | 7352 | 1 | 119 | 18099 | 0 | 0 |
(28410,7) | 12 | 2306 | 24 | |
28410 | 8 | 7232 | 1 | 117 | 18099 | 0 | 0 |
(28410,8) | 12 | 2306 | 24 | |
28410 | 9 | 7112 | 1 | 117 | 18099 | 0 | 0 |
(28410,9) | 12 | 2306 | 24 | |
28410 | 10 | 6992 | 1 | 117 | 18099 | 0 | 0 |
(28410,10) | 12 | 2306 | 24 | |
28410 | 11 | 6872 | 1 | 117 | 18099 | 0 | 0 |
(28410,11) | 12 | 2306 | 24 | |
28410 | 12 | 6752 | 1 | 117 | 18099 | 0 | 0 |
(28410,12) | 12 | 2306 | 24 | |
28410 | 13 | 6632 | 1 | 117 | 18099 | 0 | 0 |
(28410,13) | 12 | 2306 | 24 | |
28410 | 14 | 6512 | 1 | 117 | 18099 | 0 | 0 |
(28410,14) | 12 | 2306 | 24 | |
28410 | 15 | 6392 | 1 | 117 | 225753 | 225753 | 4 |
(28410,15) | 12 | 10642 | 24 | |
28410 | 16 | 6272 | 1 | 117 | 225753 | 209014 | 3 |
(30024,41) | 8204 | 12610 | 24 | |
28410 | 17 | 6152 | 1 | 117 | 225753 | 209014 | 2 |
(30024,40) | 8204 | 12610 | 24 | |
28410 | 18 | 6032 | 1 | 117 | 225753 | 209014 | 1 |
(30024,39) | 8204 | 12610 | 24 | |
28410 | 19 | 5912 | 1 | 117 | 225753 | 209014 | 0 |
(30024,38) | 8204 | 12610 | 24 | |
28410 | 20 | 5792 | 1 | 118 | 18099 | 0 | 0 |
(28410,20) | 12 | 2306 | 24 | |
28410 | 21 | 5672 | 1 | 118 | 18099 | 0 | 0 |
(28410,21) | 12 | 2306 | 24 | |
28410 | 22 | 5552 | 1 | 118 | 18099 | 0 | 0 |
(28410,22) | 12 | 2306 | 24 | |
28410 | 23 | 5432 | 1 | 118 | 18099 | 0 | 0 |
(28410,23) | 12 | 2306 | 24 | |
28410 | 24 | 5312 | 1 | 118 | 18099 | 0 | 0 |
(28410,24) | 12 | 2306 | 24 | |
28410 | 25 | 5192 | 1 | 118 | 18099 | 0 | 0 |
(28410,25) | 12 | 2306 | 24 | |
28410 | 26 | 5072 | 1 | 119 | 18099 | 0 | 0 |
(28410,26) | 12 | 2306 | 24 | |
28410 | 27 | 4952 | 1 | 118 | 18099 | 0 | 0 |
(28410,27) | 12 | 2306 | 24 | |
28410 | 28 | 4832 | 1 | 118 | 18099 | 0 | 0 |
(28410,28) | 12 | 2306 | 24 | |
28410 | 29 | 4712 | 1 | 118 | 18099 | 0 | 0 |
(28410,29) | 12 | 2306 | 24 | |
28410 | 30 | 4592 | 1 | 119 | 18099 | 0 | 0 |
(28410,30) | 12 | 2306 | 24 | |
28410 | 31 | 4472 | 1 | 118 | 18099 | 0 | 0 |
(28410,31) | 12 | 2306 | 24 | |
28410 | 32 | 4352 | 1 | 118 | 18099 | 0 | 0 |
(28410,32) | 12 | 2306 | 24 | |
28410 | 33 | 4232 | 1 | 118 | 18099 | 0 | 0 |
(28410,33) | 12 | 2306 | 24 | |
28410 | 34 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28410,34) | 12 | 2306 | 24 | |
28410 | 35 | 3992 | 1 | 117 | 18099 | 0 | 0 |
(28410,35) | 12 | 2306 | 24 | |
28410 | 36 | 3872 | 1 | 118 | 18099 | 0 | 0 |
(28410,36) | 12 | 2306 | 24 | |
28410 | 37 | 3752 | 1 | 118 | 18099 | 0 | 0 |
(28410,37) | 12 | 2306 | 24 | |
28410 | 38 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28410,38) | 12 | 2306 | 24 | |
28410 | 39 | 3512 | 1 | 117 | 18099 | 0 | 0 |
(28410,39) | 12 | 2306 | 24 | |
28410 | 40 | 3392 | 1 | 118 | 18099 | 0 | 0 |
(28410,40) | 12 | 2306 | 24 | |
28410 | 41 | 3272 | 1 | 118 | 18099 | 0 | 0 |
(28410,41) | 12 | 2306 | 24 | |
28410 | 42 | 3152 | 1 | 118 | 18099 | 0 | 0 |
(28410,42) | 12 | 2306 | 24 | |
28410 | 43 | 3032 | 1 | 118 | 18099 | 0 | 0 |
(28410,43) | 12 | 2306 | 24 | |
28410 | 44 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28410,44) | 12 | 2306 | 24 | |
28410 | 45 | 2792 | 1 | 118 | 18099 | 0 | 0 |
(28410,45) | 12 | 2306 | 24 | |
28410 | 46 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28410,46) | 12 | 2306 | 24 | |
28410 | 47 | 2552 | 1 | 118 | 18099 | 0 | 0 |
(28410,47) | 12 | 2306 | 24 | |
28410 | 48 | 2432 | 1 | 119 | 18099 | 0 | 0 |
(28410,48) | 12 | 2306 | 24 | |
28410 | 49 | 0 | 3 | 0 | | |
| | | | | |
28410 | 50 | 0 | 3 | 0 | | |
| | | | | |
28410 | 51 | 0 | 3 | 0 | | |
| | | | | |
28410 | 52 | 0 | 3 | 0 | | |
| | | | | |
28410 | 53 | 2312 | 1 | 118 | 18099 | 0 | 0 |
(28410,53) | 12 | 2306 | 24 | |
28410 | 54 | 2192 | 1 | 119 | 18099 | 0 | 0 |
(28410,54) | 12 | 2306 | 24 | |
28410 | 55 | 2072 | 1 | 118 | 18099 | 0 | 0 |
(28410,55) | 12 | 2306 | 24 | |
28410 | 56 | 1952 | 1 | 119 | 18099 | 0 | 0 |
(28410,56) | 12 | 2306 | 24 | |
28410 | 57 | 1832 | 1 | 118 | 18099 | 0 | 0 |
(28410,57) | 12 | 2306 | 24 | |
28410 | 58 | 1712 | 1 | 119 | 18099 | 0 | 0 |
(28410,58) | 12 | 2306 | 24 | |
28410 | 59 | 1592 | 1 | 118 | 18099 | 0 | 0 |
(28410,59) | 12 | 2306 | 24 | |
28410 | 60 | 1472 | 1 | 118 | 18099 | 0 | 0 |
(28410,60) | 12 | 2306 | 24 | |
28410 | 61 | 1352 | 1 | 119 | 18099 | 0 | 0 |
(28410,61) | 12 | 2306 | 24 | |
28410 | 62 | 1232 | 1 | 118 | 18099 | 0 | 0 |
(28410,62) | 12 | 2306 | 24 | |
28410 | 63 | 1112 | 1 | 119 | 18099 | 0 | 0 |
(28410,63) | 12 | 2306 | 24 | |
28410 | 64 | 992 | 1 | 118 | 18099 | 0 | 0 |
(28410,64) | 12 | 2306 | 24 | |
28410 | 65 | 872 | 1 | 118 | 18099 | 0 | 0 |
(28410,65) | 12 | 2306 | 24 | |
28561 | 1 | 8072 | 1 | 119 | 18099 | 0 | 0 |
(28561,1) | 12 | 2306 | 24 | |
28561 | 2 | 7952 | 1 | 119 | 18099 | 0 | 0 |
(28561,2) | 12 | 2306 | 24 | |
28561 | 3 | 7832 | 1 | 118 | 18099 | 0 | 0 |
(28561,3) | 12 | 2306 | 24 | |
28561 | 4 | 7712 | 1 | 118 | 18099 | 0 | 0 |
(28561,4) | 12 | 2306 | 24 | |
28561 | 5 | 7592 | 1 | 119 | 18099 | 0 | 0 |
(28561,5) | 12 | 2306 | 24 | |
28561 | 6 | 7472 | 1 | 118 | 18099 | 0 | 0 |
(28561,6) | 12 | 2306 | 24 | |
28561 | 7 | 7352 | 1 | 118 | 18099 | 0 | 0 |
(28561,7) | 12 | 2306 | 24 | |
28561 | 8 | 7232 | 1 | 119 | 18099 | 0 | 0 |
(28561,8) | 12 | 2306 | 24 | |
28561 | 9 | 7112 | 1 | 118 | 232260 | 232260 | 10 |
(28561,9) | 12 | 10642 | 24 | |
28561 | 10 | 6992 | 1 | 119 | 232260 | 232260 | 9 |
(28561,10) | 12 | 10642 | 24 | |
28561 | 11 | 6872 | 1 | 118 | 232260 | 232260 | 8 |
(28561,11) | 12 | 10642 | 24 | |
28561 | 12 | 6752 | 1 | 119 | 232260 | 232260 | 7 |
(28561,12) | 12 | 10642 | 24 | |
28561 | 13 | 6632 | 1 | 118 | 232260 | 232260 | 6 |
(28561,13) | 12 | 10642 | 24 | |
28561 | 14 | 6512 | 1 | 119 | 232260 | 232260 | 5 |
(28561,14) | 12 | 10642 | 24 | |
28561 | 15 | 6392 | 1 | 118 | 232260 | 232260 | 4 |
(28561,15) | 12 | 10642 | 24 | |
28561 | 16 | 6272 | 1 | 119 | 232260 | 269002 | 0 |
(30200,31) | 8204 | 14658 | 24 | |
28561 | 17 | 6152 | 1 | 118 | 232260 | 269002 | 0 |
(30200,30) | 8204 | 14658 | 24 | |
28561 | 18 | 6032 | 1 | 119 | 232260 | 269002 | 0 |
(30200,29) | 8204 | 6466 | 24 | |
28561 | 19 | 5912 | 1 | 118 | 232260 | 269002 | 0 |
(30200,28) | 8204 | 6466 | 24 | |
28561 | 20 | 5792 | 1 | 118 | 18099 | 0 | 0 |
(28561,20) | 12 | 2306 | 24 | |
28561 | 21 | 5672 | 1 | 118 | 18099 | 0 | 0 |
(28561,21) | 12 | 2306 | 24 | |
28561 | 22 | 5552 | 1 | 119 | 18099 | 0 | 0 |
(28561,22) | 12 | 2306 | 24 | |
28561 | 23 | 5432 | 1 | 118 | 18099 | 0 | 0 |
(28561,23) | 12 | 2306 | 24 | |
28561 | 24 | 5312 | 1 | 118 | 18099 | 0 | 0 |
(28561,24) | 12 | 2306 | 24 | |
28561 | 25 | 5192 | 1 | 118 | 18099 | 0 | 0 |
(28561,25) | 12 | 2306 | 24 | |
28561 | 26 | 5072 | 1 | 118 | 18099 | 0 | 0 |
(28561,26) | 12 | 2306 | 24 | |
28561 | 27 | 4952 | 1 | 118 | 18099 | 0 | 0 |
(28561,27) | 12 | 2306 | 24 | |
28561 | 28 | 4832 | 1 | 118 | 18099 | 0 | 0 |
(28561,28) | 12 | 2306 | 24 | |
28561 | 29 | 4712 | 1 | 118 | 18099 | 0 | 0 |
(28561,29) | 12 | 2306 | 24 | |
28561 | 30 | 4592 | 1 | 118 | 18099 | 0 | 0 |
(28561,30) | 12 | 2306 | 24 | |
28561 | 31 | 4472 | 1 | 118 | 18099 | 0 | 0 |
(28561,31) | 12 | 2306 | 24 | |
28561 | 32 | 4352 | 1 | 118 | 18099 | 0 | 0 |
(28561,32) | 12 | 2306 | 24 | |
28561 | 33 | 4232 | 1 | 118 | 18099 | 0 | 0 |
(28561,33) | 12 | 2306 | 24 | |
28561 | 34 | 4112 | 1 | 118 | 18099 | 0 | 0 |
(28561,34) | 12 | 2306 | 24 | |
28561 | 35 | 3992 | 1 | 119 | 18099 | 0 | 0 |
(28561,35) | 12 | 2306 | 24 | |
28561 | 36 | 3872 | 1 | 118 | 18099 | 0 | 0 |
(28561,36) | 12 | 2306 | 24 | |
28561 | 37 | 3752 | 1 | 118 | 18099 | 0 | 0 |
(28561,37) | 12 | 2306 | 24 | |
28561 | 38 | 3632 | 1 | 118 | 18099 | 0 | 0 |
(28561,38) | 12 | 2306 | 24 | |
28561 | 39 | 3512 | 1 | 119 | 18099 | 0 | 0 |
(28561,39) | 12 | 2306 | 24 | |
28561 | 40 | 3392 | 1 | 119 | 18099 | 0 | 0 |
(28561,40) | 12 | 2306 | 24 | |
28561 | 41 | 3272 | 1 | 119 | 18099 | 0 | 0 |
(28561,41) | 12 | 2306 | 24 | |
28561 | 42 | 3152 | 1 | 118 | 18099 | 0 | 0 |
(28561,42) | 12 | 2306 | 24 | |
28561 | 43 | 3032 | 1 | 118 | 18099 | 0 | 0 |
(28561,43) | 12 | 2306 | 24 | |
28561 | 44 | 2912 | 1 | 118 | 18099 | 0 | 0 |
(28561,44) | 12 | 2306 | 24 | |
28561 | 45 | 2792 | 1 | 118 | 18099 | 0 | 0 |
(28561,45) | 12 | 2306 | 24 | |
28561 | 46 | 2672 | 1 | 118 | 18099 | 0 | 0 |
(28561,46) | 12 | 2306 | 24 | |
28561 | 47 | 2552 | 1 | 118 | 18099 | 0 | 0 |
(28561,47) | 12 | 2306 | 24 | |
28561 | 48 | 2432 | 1 | 119 | 18099 | 0 | 0 |
(28561,48) | 12 | 2306 | 24 | |
28561 | 49 | 2312 | 1 | 118 | 18099 | 0 | 0 |
(28561,49) | 12 | 2306 | 24 | |
28561 | 50 | 2192 | 1 | 118 | 18099 | 0 | 0 |
(28561,50) | 12 | 2306 | 24 | |
28561 | 51 | 2072 | 1 | 119 | 18099 | 0 | 0 |
(28561,51) | 12 | 2306 | 24 | |
28561 | 52 | 1952 | 1 | 119 | 18099 | 0 | 0 |
(28561,52) | 12 | 2306 | 24 | |
28561 | 53 | 1832 | 1 | 118 | 18099 | 0 | 0 |
(28561,53) | 12 | 2306 | 24 | |
28561 | 54 | 1712 | 1 | 118 | 18099 | 0 | 0 |
(28561,54) | 12 | 2306 | 24 | |
28561 | 55 | 1592 | 1 | 119 | 18099 | 0 | 0 |
(28561,55) | 12 | 2306 | 24 | |
28561 | 56 | 1472 | 1 | 118 | 232282 | 232282 | 5 |
(28561,56) | 12 | 10642 | 24 | |
28561 | 57 | 1352 | 1 | 119 | 232282 | 232282 | 4 |
(28561,57) | 12 | 10642 | 24 | |
28561 | 58 | 1232 | 1 | 118 | 232282 | 232282 | 3 |
(28561,58) | 12 | 10642 | 24 | |
28561 | 59 | 1112 | 1 | 118 | 232282 | 232282 | 2 |
(28561,59) | 12 | 10642 | 24 | |
28561 | 60 | 992 | 1 | 118 | 232282 | 232282 | 1 |
(28561,60) | 12 | 10642 | 24 | |
28561 | 61 | 872 | 1 | 119 | 232282 | 232282 | 0 |
(28561,61) | 12 | 10642 | 24 | |
28561 | 62 | 752 | 1 | 118 | 18099 | 0 | 0 |
(28561,62) | 12 | 2306 | 24 | |
28561 | 63 | 632 | 1 | 118 | 18099 | 0 | 0 |
(28561,63) | 12 | 2306 | 24 | |
28561 | 64 | 512 | 1 | 118 | 18099 | 0 | 0 |
(28561,64) | 12 | 2306 | 24 | |
28561 | 65 | 392 | 1 | 119 | 18099 | 0 | 0 |
(28561,65) | 12 | 2306 | 24 | |
(602 rows)
Just wanted to check in here: is there any other info I need to provide? Any conclusions about this instance? Thanks, Maciek
On 2013-12-06 09:23:59 -0800, Maciek Sakrejda wrote:
> => with affected_pages as (
> select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
> from post where id in (select id from post group by id having count(*) >
> 1)
> ) select * from affected_pages, page_header(get_raw_page('post',
> page::integer));
> page | lsn | checksum | flags | lower | upper | special |
> pagesize | version | prune_xid
> -------+-------------+----------+-------+-------+-------+---------+----------+---------+-----------
> 28192 | 12/1B004250 | -1751 | 1 | 304 | 1712 | 8192 |
> 8192 | 4 | 0
> 28194 | 12/1D1304D8 | 10886 | 1 | 316 | 1352 | 8192 |
> 8192 | 4 | 0
> 28203 | 12/BE2BDF8 | -12970 | 0 | 284 | 1592 | 8192 |
> 8192 | 4 | 0
> 28204 | 12/BE2DDE0 | 1235 | 0 | 284 | 392 | 8192 |
> 8192 | 4 | 0
> 28281 | 12/BE33218 | -7837 | 0 | 284 | 392 | 8192 |
> 8192 | 4 | 0
> 28390 | 12/BE4CF70 | -20573 | 0 | 284 | 392 | 8192 |
> 8192 | 4 | 0
> 28392 | 12/1D135E78 | -8032 | 1 | 300 | 992 | 8192 |
> 8192 | 4 | 0
> 28410 | 12/BE67610 | 23108 | 0 | 284 | 872 | 8192 |
> 8192 | 4 | 0
> 28561 | 12/1D164030 | -24596 | 0 | 284 | 392 | 8192 |
> 8192 | 4 | 0
> (9 rows)
>
> => with affected_pages as (
> select distinct regexp_replace(ctid::text, '\((\d+),\d+\)', '\1') as page
> from post where id in (select id from post group by id having count(*) >
> 1)
> ) select * from affected_pages, heap_page_items(get_raw_page('post',
> page::integer));
> page | lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 |
> t_ctid | t_infomask2 | t_infomask | t_hoff | t_bits | t_oid
>
-------+----+--------+----------+--------+--------+--------+----------+------------+-------------+------------+--------+--------+-------
> 28192 | 1 | 8072 | 1 | 118 | 18099 | 0 | 0 |
> (28192,1) | 12 | 2306 | 24 | |
> 28192 | 2 | 7952 | 1 | 118 | 18099 | 80873 | 13 |
> (28192,2) | 12 | 6466 | 24 | |
> 28192 | 3 | 7832 | 1 | 118 | 18099 | 80873 | 12 |
> (28192,3) | 12 | 6466 | 24 | |
> 28192 | 4 | 0 | 3 | 0 | | |
To me this pretty clearly indicates the multixact vacuuming bug from
9.3.2. But given your pg_controldata output:
Latest checkpoint's NextXID: 0/1579944
Latest checkpoint's NextMultiXactId: 592631
Latest checkpoint's NextMultiOffset: 1236955
with of ~600k multis used so far, I cannot see how it could have been
triggered without either nondefault vacuum settings or a vacuum freeze.
Do you perhaps automatedly run VACUUM with different settings from
crontab or similar? Or have per-table vacuum settings?
Greetings,
Andres Freund
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
Interesting. And that was the pg_controldata of the live database. The PITR, which is a few days older, has a much lower NextMultiXactId (and I'm including everything else, since I'm not sure what's relevant and what isn't): pg_control version number: 937 Catalog version number: 201306121 Database system identifier: 5951409877055172578 Database cluster state: in production pg_control last modified: Tue 10 Dec 2013 01:30:39 AM UTC Latest checkpoint location: 14/6E000060 Prior checkpoint location: 14/6C000060 Latest checkpoint's REDO location: 14/6E000028 Latest checkpoint's REDO WAL file: 00000002000000140000006E Latest checkpoint's TimeLineID: 2 Latest checkpoint's PrevTimeLineID: 2 Latest checkpoint's full_page_writes: on Latest checkpoint's NextXID: 0/715897 Latest checkpoint's NextOID: 655360 Latest checkpoint's NextMultiXactId: 269003 Latest checkpoint's NextMultiOffset: 556707 Latest checkpoint's oldestXID: 1845 Latest checkpoint's oldestXID's DB: 1 Latest checkpoint's oldestActiveXID: 715897 Latest checkpoint's oldestMultiXid: 1 Latest checkpoint's oldestMulti's DB: 1 Time of latest checkpoint: Tue 10 Dec 2013 01:30:38 AM UTC Fake LSN counter for unlogged rels: 0/1 Minimum recovery ending location: 0/0 Min recovery ending loc's timeline: 0 Backup start location: 0/0 Backup end location: 0/0 End-of-backup record required: no Current wal_level setting: hot_standby Current max_connections setting: 500 Current max_prepared_xacts setting: 0 Current max_locks_per_xact setting: 64 Maximum data alignment: 8 Database block size: 8192 Blocks per segment of large relation: 131072 WAL block size: 8192 Bytes per WAL segment: 16777216 Maximum length of identifiers: 64 Maximum columns in an index: 32 Maximum size of a TOAST chunk: 1996 Date/time type storage: 64-bit integers Float4 argument passing: by value Float8 argument passing: by value Data page checksum version: 1 We don't schedule vacuum specially: we just depend on autovacuum. Our pg_restore wrapper was used to set up this database, so I know we ran ANALYZE (on the whole DB) after the data load, but that's it. The user is quite certain that he has never run VACUUM FREEZE on this instance (and we did not do so either). And no special settings exist for that table (in fact, reloptions is null for every table in pg_class--or is there somewhere else I might need to look?). Could there be other edge cases having to do with a brand-new instance? I think I mentioned that initdb was just a day or so before the error was first noticed.
Maciek Sakrejda <maciek@heroku.com> schrieb: >By the way, the customer has run into the issue *again* now since >deleting >the duplicate rows about a week ago. There are three duplicates now. >Again, >everything is identical between the versions except for the ctid. The >xmin/xmax are also identical: > >=> select ctid, xmin, xmax, id from post where id in (1877554, 1885823, >1885824); > ctid | xmin | xmax | id >------------+---------+---------+--------- > (28771,35) | 1651525 | 1651525 | 1877554 > (28778,29) | 1651525 | 1651525 | 1877554 > (28771,37) | 1651525 | 1651525 | 1885823 > (28778,33) | 1651525 | 1651525 | 1885823 > (28771,36) | 1651525 | 1651525 | 1885824 > (28778,31) | 1651525 | 1651525 | 1885824 >(6 rows) Have you upgraded to 9.3.2? Andres -- Please excuse brevity and formatting - I am writing this on my mobile phone. Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
By the way, the customer has run into the issue *again* now since deleting
the duplicate rows about a week ago. There are three duplicates now. Again,
everything is identical between the versions except for the ctid. The
xmin/xmax are also identical:
=> select ctid, xmin, xmax, id from post where id in (1877554, 1885823,
1885824);
ctid | xmin | xmax | id
------------+---------+---------+---------
(28771,35) | 1651525 | 1651525 | 1877554
(28778,29) | 1651525 | 1651525 | 1877554
(28771,37) | 1651525 | 1651525 | 1885823
(28778,33) | 1651525 | 1651525 | 1885823
(28771,36) | 1651525 | 1651525 | 1885824
(28778,31) | 1651525 | 1651525 | 1885824
(6 rows)
No, not yet--just providing another data point. I understand bug may recur before patching.
Last night, I ran into an issue on a promoted 9.3.2 replica from this
instance (the original was a 9.3.1, but no 9.3.1 replica was ever promoted
from it, just 9.3.2), and a few folks on IRC were incredibly helpful in
debugging the issue (thanks again), but I don't think we came to any clear
conclusion. The final verdict was that a dump/restore is the best bet and
I'm working with the customer to do that, but I want to make sure that the
issues here fit the symptoms of the bugs fixed in 9.3.2, and that there's
nothing else lurking that may bite us in the future. Based on the earlier
comments in this thread, I got the impression that that's not quite the
case. I'm happy to dig in further if that's useful, but I'm not sure what
I'm looking for.
Anyway, the symptoms last night were basically intermittent failures in
creating temporary tables. I ran a statement like:
=> CREATE TEMPORARY TABLE foo(id INTEGER NOT NULL, bar INTEGER, PRIMARY KEY
(id)) ON COMMIT DROP;
and occasionally this would work, but sometimes yield errors like:
ERROR: could not read block 20 in file "base/16385/12661": read only 0 of
8192 bytes
12661 is pg_type, and the other table that showed up in the error is
pg_constraint. Block 20 here is beyond the end of the table
(pg_relation_size was 155648), and I got similar errors for pg_constraint.
The error was intermittent (sometimes creating the table worked fine) and
the block that could not be read was one to three blocks beyond the end of
the table. I did not see any other errors.
Based on guidance in IRC, I checked the raw page information for the tables
and all their indexes with the pageinspect extension,
select i.* from generate_series(0,18) bn, get_raw_page('pg_type', bn) p,
heap_page_items(p) i;
select b.* from generate_series(1,3) bn, bt_page_items('pg_type_oid_index',
bn) b;
and did not see any blocks outside of the expected range referenced there.
I also checked the FSM for both tables with the pg_freespacemap extension:
select * from pg_freespace('pg_type');
but did not find any blocks outside of the expected range for either.
I also checked relfrozenxid, relminmxid in pg_class for the tables showing
errors:
relname | relfrozenxid | relminmxid
---------------+--------------+------------
pg_type | 1845 | 1
pg_constraint | 1845 | 1
select txid_current() was 4011697 at the time.
I think this is all the relevant info from the IRC session. As before,
there was no explicit VACUUM FREEZE and no changes to autovacuum settings.
So, does this fit the bill for the issues fixed in 9.3.2? As I understand
it, the fact that the current txid is much lower than the wraparound
threshold suggests no, right? If not, are there more diagnostics I can run
to learn anything here? If there's another bug to find here, I'd love to
help find it.
Thanks,
Maciek
On Wed, Dec 4, 2013 at 8:04 PM, <maciek@heroku.com> wrote: > "post_col2_col3_idx" btree (col2, col3) CLUSTER Andres, would cluster trigger the same MultiXactId freezing problem? -- greg
On 2013-12-20 00:19:16 +0000, Greg Stark wrote: > On Wed, Dec 4, 2013 at 8:04 PM, <maciek@heroku.com> wrote: > > "post_col2_col3_idx" btree (col2, col3) CLUSTER > > Andres, would cluster trigger the same MultiXactId freezing problem? Yes, but IIRC, it wouldn't trigger the problems in this specific cluster unless somebody changed vacuum_freeze_min_age since the cluster was younger than that when the problem first occurered. Check vacuum_set_xid_limits() for details. Nice observation about the CLUSTER in there. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services