Thread: BUG #15142: ERROR: MultiXactId nnnnn has not been created yet --apparent wraparound in v9.5

The following bug has been logged on the website:

Bug reference:      15142
Logged by:          Александр Молофеев
Email address:      molofeev3@gmail.com
PostgreSQL version: 9.5.12
Operating system:   Ubuntu 16.04
Description:

My version of software:  PostgreSQL 9.5.12 on x86_64-pc-linux-gnu, compiled
by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, 64-bit
When I run updating a table(table size ~100Gb) I got an error.
For example, vacuum analyze command output:
ERROR:  MultiXactId 7629921 has not been created yet -- apparent
wraparound
Autovacuum is on.

PG Bug reporting form wrote:

> My version of software:  PostgreSQL 9.5.12 on x86_64-pc-linux-gnu, compiled
> by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, 64-bit
> When I run updating a table(table size ~100Gb) I got an error.
> For example, vacuum analyze command output:
> ERROR:  MultiXactId 7629921 has not been created yet -- apparent
> wraparound
> Autovacuum is on.

Can you please show the output of pg_controldata in this directory?
Also:
 select datminmxid from pg_database where datname = current_database();
for the database where the error occurred.

Thanks

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Thank you for reply.
Output of pg_controldata:

pg_control version number:            942
Catalog version number:               201510051
Database system identifier:           6520087809676833048
Database cluster state:               in production
pg_control last modified:             вт, 03-кві-2018 14:43:46 +0300
Latest checkpoint location:           327/D0544BA0
Prior checkpoint location:            327/D03F3D88
Latest checkpoint's REDO location:    327/D0544BA0
Latest checkpoint's REDO WAL file:    0000000100000327000000D0
Latest checkpoint's TimeLineID:       1
Latest checkpoint's PrevTimeLineID:   1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID:          0/1564
Latest checkpoint's NextOID:          10750280
Latest checkpoint's NextMultiXactId:  1
Latest checkpoint's NextMultiOffset:  0
Latest checkpoint's oldestXID:        615
Latest checkpoint's oldestXID's DB:   1
Latest checkpoint's oldestActiveXID:  0
Latest checkpoint's oldestMultiXid:   1
Latest checkpoint's oldestMulti's DB: 1
Latest checkpoint's oldestCommitTsXid:0
Latest checkpoint's newestCommitTsXid:0
Time of latest checkpoint:            вт, 03-кві-2018 14:43:23 +0300
Fake LSN counter for unlogged rels:   0/1
Minimum recovery ending location:     0/0
Min recovery ending loc's timeline:   0
Backup start location:                0/0
Backup end location:                  0/0
End-of-backup record required:        no
wal_level setting:                    minimal
wal_log_hints setting:                off
max_connections setting:              100
max_worker_processes setting:         8
max_prepared_xacts setting:           0
max_locks_per_xact setting:           64
track_commit_timestamp setting:       off
Maximum data alignment:               8
Database block size:                  8192
Blocks per segment of large relation: 131072
WAL block size:                       8192
Bytes per WAL segment:                16777216
Maximum length of identifiers:        64
Maximum columns in an index:          32
Maximum size of a TOAST chunk:        1996
Size of a large-object chunk:         2048
Date/time type storage:               64-bit integers
Float4 argument passing:              by value
Float8 argument passing:              by value
Data page checksum version:           0


Output of [select datminmxid from pg_database where datname = 
current_database();]:

datminmxid
------------
           1

> PG Bug reporting form wrote:
>
>> My version of software:  PostgreSQL 9.5.12 on x86_64-pc-linux-gnu, compiled
>> by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609, 64-bit
>> When I run updating a table(table size ~100Gb) I got an error.
>> For example, vacuum analyze command output:
>> ERROR:  MultiXactId 7629921 has not been created yet -- apparent
>> wraparound
>> Autovacuum is on.
> Can you please show the output of pg_controldata in this directory?
> Also:
>   select datminmxid from pg_database where datname = current_database();
> for the database where the error occurred.
>
> Thanks
>

molofeev wrote:
> Thank you for reply.
> Output of pg_controldata:
> 
> Latest checkpoint's NextMultiXactId:  1
> Latest checkpoint's NextMultiOffset:  0

Uh, this says that this cluster has never used any multixacts.

> Output of [select datminmxid from pg_database where datname =
> current_database();]:
> 
> datminmxid
> ------------
>           1

... and this confirms.

I think the easiest way forward is to identify the problem page and see
what's there.  Please find what command was being run that caused the
error, which might help narrow down which table it is; then scan the
table to see on which page you can see the error.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

The last query that working is - select * from TABLE_NAME limit 1 offset 
160;

the next query

select * from TABLE_NAME limit 1 offset 162;

return - ERROR: MultiXactId 1048586 has not been created yet -- apparent 
wraparound
SQL state: XX000


On 04/03/2018 04:23 PM, Alvaro Herrera wrote:
> molofeev wrote:
>> Thank you for reply.
>> Output of pg_controldata:
>>
>> Latest checkpoint's NextMultiXactId:  1
>> Latest checkpoint's NextMultiOffset:  0
> Uh, this says that this cluster has never used any multixacts.
>
>> Output of [select datminmxid from pg_database where datname =
>> current_database();]:
>>
>> datminmxid
>> ------------
>>            1
> ... and this confirms.
>
> I think the easiest way forward is to identify the problem page and see
> what's there.  Please find what command was being run that caused the
> error, which might help narrow down which table it is; then scan the
> table to see on which page you can see the error.
>

Any ideas?


On 04/03/2018 04:36 PM, molofeev wrote:
> The last query that working is - select * from TABLE_NAME limit 1 
> offset 160;
>
> the next query
>
> select * from TABLE_NAME limit 1 offset 162;
>
> return - ERROR: MultiXactId 1048586 has not been created yet -- 
> apparent wraparound
> SQL state: XX000
>
>
> On 04/03/2018 04:23 PM, Alvaro Herrera wrote:
>> molofeev wrote:
>>> Thank you for reply.
>>> Output of pg_controldata:
>>>
>>> Latest checkpoint's NextMultiXactId:  1
>>> Latest checkpoint's NextMultiOffset:  0
>> Uh, this says that this cluster has never used any multixacts.
>>
>>> Output of [select datminmxid from pg_database where datname =
>>> current_database();]:
>>>
>>> datminmxid
>>> ------------
>>>            1
>> ... and this confirms.
>>
>> I think the easiest way forward is to identify the problem page and see
>> what's there.  Please find what command was being run that caused the
>> error, which might help narrow down which table it is; then scan the
>> table to see on which page you can see the error.
>>
>

On 04/04/2018 10:20 AM, molofeev wrote:
> Any ideas?
> 
> 
> On 04/03/2018 04:36 PM, molofeev wrote:
>> The last query that working is - select * from TABLE_NAME limit 1
>> offset 160;
>>
>> the next query
>>
>> select * from TABLE_NAME limit 1 offset 162;
>>
>> return - ERROR: MultiXactId 1048586 has not been created yet --
>> apparent wraparound
>> SQL state: XX000
>>

Well, as Alvaro mentioned, we need to see the page that contains the
record with that multixactid. You now know which table it is, and you
know it's tuple #162. We still don't know which page is it exactly, but
you can determine that by using CTID:

   select * from TABLE_NAME where ctid between '(13,1)'::tid
                              and '(14,0)'::tid;

You'll need to start on page 0 and increase it until you get the error
again. This will scan the whole table, though.

Then you know which page has the broken data, and you can inspect it
using pageinspect - extract it using raw_page, inspect it using various
functions in that extension. I don't know how sensitive the data in that
table is, but perhaps you may share the 8kB page.

FWIW this seems to be some sort of data corruption, where the XID got
overwritten by a bogus value in some way. The bigger question is how
many other such cases are there.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Thank you for reply.

> The bigger question is how
> many other such cases are there.
I can get data from table only before tuple #162, after that all queries 
(for example getting tuple 163, 164 ... etc) return error.


Table contain openstreatmap data, maybe errors occurs because when i was 
creating necessary indexes for table, happened server crash and I 
relaunch indexes creation.

Query: select * from TABLE_NAME where ctid between '(0,1)'::tid and 
'(1,1)'::tid; return ERROR:  MultiXactId 1048586 has not been created 
yet -- apparent wraparound


Tuple #161 has ctid = (3220145, 81)



On 04/04/2018 12:22 PM, Tomas Vondra wrote:
> On 04/04/2018 10:20 AM, molofeev wrote:
>> Any ideas?
>>
>>
>> On 04/03/2018 04:36 PM, molofeev wrote:
>>> The last query that working is - select * from TABLE_NAME limit 1
>>> offset 160;
>>>
>>> the next query
>>>
>>> select * from TABLE_NAME limit 1 offset 162;
>>>
>>> return - ERROR: MultiXactId 1048586 has not been created yet --
>>> apparent wraparound
>>> SQL state: XX000
>>>
> Well, as Alvaro mentioned, we need to see the page that contains the
> record with that multixactid. You now know which table it is, and you
> know it's tuple #162. We still don't know which page is it exactly, but
> you can determine that by using CTID:
>
>     select * from TABLE_NAME where ctid between '(13,1)'::tid
>                                and '(14,0)'::tid;
>
> You'll need to start on page 0 and increase it until you get the error
> again. This will scan the whole table, though.
>
> Then you know which page has the broken data, and you can inspect it
> using pageinspect - extract it using raw_page, inspect it using various
> functions in that extension. I don't know how sensitive the data in that
> table is, but perhaps you may share the 8kB page.
>
> FWIW this seems to be some sort of data corruption, where the XID got
> overwritten by a bogus value in some way. The bigger question is how
> many other such cases are there.
>
> regards
>

>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:

 molofeev> Tuple #161 has ctid = (3220145, 81)

try these (after installing the pageinspect extension):

select * from heap_page_items(get_raw_page('tablename',3220145));

select * from heap_page_items(get_raw_page('tablename',3220146));

If you catch me on the irc channel (#postgresql on freenode.net), I
often talk people through this kind of analysis in real time.

-- 
Andrew (irc:RhodiumToad)

Thank you for reply.
First query result has data in each cell except last column.
Second query result has data only in this columns: lp, lp_off, lp_flags, lp_len;  other cells empty.
If you need I can send files with query results.



On 04/04/2018 04:53 PM, Andrew Gierth wrote:
>>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:
>   molofeev> Tuple #161 has ctid = (3220145, 81)
>
> try these (after installing the pageinspect extension):
>
> select * from heap_page_items(get_raw_page('tablename',3220145));
>
> select * from heap_page_items(get_raw_page('tablename',3220146));
>
> If you catch me on the irc channel (#postgresql on freenode.net), I
> often talk people through this kind of analysis in real time.
>

>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:

 molofeev> Thank you for reply.
 
 molofeev> First query result has data in each cell except last column.
 molofeev> Second query result has data only in this columns: lp,
 molofeev> lp_off, lp_flags, lp_len; other cells empty. If you need I
 molofeev> can send files with query results.

Please send the results, yes. Also try the same query for larger block
numbers until you find one with more data, and send that result too.

-- 
Andrew (irc:RhodiumToad)

Query 1:  select * from heap_page_items(get_raw_page('tablename',3220145));

Query 2: select * from heap_page_items(get_raw_page('tablename',3220146));

Query 3: select * from heap_page_items(get_raw_page('tablename',3220147));

Files with same names (query1, query2, query3) in attachment archive.


On 04/04/2018 05:45 PM, Andrew Gierth wrote:
>>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:
>   molofeev> Thank you for reply.
>   
>   molofeev> First query result has data in each cell except last column.
>   molofeev> Second query result has data only in this columns: lp,
>   molofeev> lp_off, lp_flags, lp_len; other cells empty. If you need I
>   molofeev> can send files with query results.
>
> Please send the results, yes. Also try the same query for larger block
> numbers until you find one with more data, and send that result too.
>

>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:

 molofeev> Query 2: select * from heap_page_items(get_raw_page('tablename',3220146));

This definitely looks corrupt. Can you do:

select encode(get_raw_page('tablename',3220146),'base64');

and send the result of that?

-- 
Andrew (irc:RhodiumToad)

molofeev wrote:
> Query 1:  select * from heap_page_items(get_raw_page('tablename',3220145));

In this one, t_infomask is 11011 for all the tuples, which is 0x2B03.
HEAP_XMAX_IS_MULTI is 0x1000.  I think you're looking at the wrong
pages, probably because of synchronized_seqscans.

I didn't look at the other pages.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>>>>> "Alvaro" == Alvaro Herrera <alvherre@alvh.no-ip.org> writes:

 >> Query 1:  select * from heap_page_items(get_raw_page('tablename',3220145));

 Alvaro> In this one, t_infomask is 11011 for all the tuples, which is
 Alvaro> 0x2B03. HEAP_XMAX_IS_MULTI is 0x1000. I think you're looking at
 Alvaro> the wrong pages, probably because of synchronized_seqscans.

3220145 appears to be a valid page (it's the one with the last
selectable row in the last slot). It's 3220146 which looks corrupt, but
it's corrupt in such a way that heap_page_items doesn't report anything
interesting because all the lp_len fields are 16.

My working hypothesis is that page 3220146 has actually been overwritten
by a page from some index, which is why it looks like valid (but too
short) items. The multixact error occurs because pg is reading garbage
from the xid and infomask fields since those don't exist in index
tuples.

--
Andrew (irc:RhodiumToad)

Andrew Gierth wrote:
> >>>>> "Alvaro" == Alvaro Herrera <alvherre@alvh.no-ip.org> writes:
> 
>  >> Query 1:  select * from heap_page_items(get_raw_page('tablename',3220145));
> 
>  Alvaro> In this one, t_infomask is 11011 for all the tuples, which is
>  Alvaro> 0x2B03. HEAP_XMAX_IS_MULTI is 0x1000. I think you're looking at
>  Alvaro> the wrong pages, probably because of synchronized_seqscans.
> 
> 3220145 appears to be a valid page (it's the one with the last
> selectable row in the last slot).

Ah.

> It's 3220146 which looks corrupt, but it's corrupt in such a way that
> heap_page_items doesn't report anything interesting because all the
> lp_len fields are 16.

/me scratches head

> My working hypothesis is that page 3220146 has actually been overwritten
> by a page from some index, which is why it looks like valid (but too
> short) items.

I guess that'd also explain why there are so many tuples -- beyond the
maximum for heap pages.

> The multixact error occurs because pg is reading garbage from the xid
> and infomask fields since those don't exist in index tuples.

Ouch.  I had never seen this.  Maybe the OP can ship the page image?
May contain private data, though.

Maybe try to read with with bt_page_items() and/or other index methods
to confirm the theory?

I would guess that this is some sort of storage error -- a SAN or RAID
controller that malfunctions, or something like that.  This I've seen.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

>>>>> "Andrew" == Andrew Gierth <andrew@tao11.riddles.org.uk> writes:

 Andrew> My working hypothesis is that page 3220146 has actually been
 Andrew> overwritten by a page from some index, which is why it looks
 Andrew> like valid (but too short) items. The multixact error occurs
 Andrew> because pg is reading garbage from the xid and infomask fields
 Andrew> since those don't exist in index tuples.

So it looks like I was partially correct.

Below is the hexdump of page 3220146 as text. Here are the notable
features:

1. The block is clearly torn at offset 0x1000 (4096 bytes). This is an
OS page or a disk sector boundary, which (in my view) points quite
strongly to this being an OS or hardware issue rather than a postgresql
bug.

You said a server crash occurred while you were creating indexes. Was
that a hard crash (power failure, etc.) and is it possible that you had
fsync disabled in postgresql.conf, or hardware or filesystems that don't
respect fsync properly?

2. The second half of the block contains tuple data which is plausibly
consistent with the valid tuples from the previous block (similar xids
and data lengths).

3. The first half of the block contains data which is clearly from some
index. But I don't think this fact is significant; when I've looked at
these kinds of corruption cases the intruding data has often come from
some file not related to postgres at all, so it's quite likely that it's
pure chance or circumstance that it happens to be valid postgres index
data in this case.

4. Obviously the data in the corrupted block is not easily recoverable.
If you need to recover the remaining data in the table, it could
probably best be done by zeroing only this specific block. Whether this
is worth doing will depend on whether it would be easier to just
recreate the data.


00000000  1c 02 00 00 48 f2 42 ed  00 00 00 00 ec 05 a0 08  |....H.B.........|
00000010  f0 1f 04 20 00 00 00 00  e0 9f 20 00 d0 9f 20 00  |... ...... ... .|
00000020  c0 9f 20 00 a0 88 20 00  b0 9f 20 00 a0 9f 20 00  |.. ... ... ... .|
00000030  d0 88 20 00 90 9f 20 00  80 9f 20 00 00 8f 20 00  |.. ... ... ... .|
00000040  70 9f 20 00 20 8f 20 00  60 9f 20 00 60 8f 20 00  |p. . . .`. .`. .|
00000050  50 9f 20 00 b0 88 20 00  40 9f 20 00 30 9f 20 00  |P. ... .@. .0. .|
00000060  40 8f 20 00 20 9f 20 00  c0 88 20 00 10 9f 20 00  |@. . . ... ... .|
00000070  00 9f 20 00 30 8f 20 00  f0 9e 20 00 50 8f 20 00  |.. .0. ... .P. .|
00000080  e0 9e 20 00 f0 8a 20 00  d0 9e 20 00 50 8c 20 00  |.. ... ... .P. .|
00000090  c0 9e 20 00 70 8f 20 00  b0 9e 20 00 50 90 20 00  |.. .p. ... .P. .|
000000a0  a0 9e 20 00 60 90 20 00  90 9e 20 00 40 8c 20 00  |.. .`. ... .@. .|
000000b0  80 9e 20 00 00 90 20 00  70 9e 20 00 10 90 20 00  |.. ... .p. ... .|
000000c0  60 9e 20 00 20 90 20 00  50 9e 20 00 e0 8f 20 00  |`. . . .P. ... .|
000000d0  40 9e 20 00 f0 8f 20 00  30 9e 20 00 40 90 20 00  |@. ... .0. .@. .|
000000e0  20 9e 20 00 90 8f 20 00  10 9e 20 00 80 8f 20 00  | . ... ... ... .|
000000f0  00 9e 20 00 a0 8f 20 00  f0 9d 20 00 c0 8f 20 00  |.. ... ... ... .|
00000100  e0 9d 20 00 b0 8f 20 00  d0 9d 20 00 30 90 20 00  |.. ... ... .0. .|
00000110  c0 9d 20 00 10 8f 20 00  b0 9d 20 00 d0 8f 20 00  |.. ... ... ... .|
00000120  a0 9d 20 00 70 90 20 00  90 9d 20 00 a0 90 20 00  |.. .p. ... ... .|
00000130  80 9d 20 00 90 90 20 00  70 9d 20 00 80 90 20 00  |.. ... .p. ... .|
00000140  60 9d 20 00 20 8c 20 00  50 9d 20 00 c0 90 20 00  |`. . . .P. ... .|
00000150  40 9d 20 00 b0 90 20 00  30 9d 20 00 e0 90 20 00  |@. ... .0. ... .|
00000160  20 9d 20 00 d0 90 20 00  10 9d 20 00 a0 8e 20 00  | . ... ... ... .|
00000170  00 9d 20 00 b0 8e 20 00  f0 9c 20 00 f0 8e 20 00  |.. ... ... ... .|
00000180  e0 9c 20 00 90 8e 20 00  d0 9c 20 00 e0 8e 20 00  |.. ... ... ... .|
00000190  c0 9c 20 00 c0 8e 20 00  b0 9c 20 00 60 8e 20 00  |.. ... ... .`. .|
000001a0  a0 9c 20 00 d0 8e 20 00  90 9c 20 00 50 8e 20 00  |.. ... ... .P. .|
000001b0  80 9c 20 00 00 8e 20 00  70 9c 20 00 f0 8d 20 00  |.. ... .p. ... .|
000001c0  60 9c 20 00 10 8e 20 00  50 9c 20 00 30 8e 20 00  |`. ... .P. .0. .|
000001d0  40 9c 20 00 20 8e 20 00  30 9c 20 00 40 8e 20 00  |@. . . .0. .@. .|
000001e0  20 9c 20 00 90 8d 20 00  10 9c 20 00 00 9c 20 00  | . ... ... ... .|
000001f0  f0 9b 20 00 e0 9b 20 00  d0 9b 20 00 80 91 20 00  |.. ... ... ... .|
00000200  c0 9b 20 00 40 8b 20 00  b0 9b 20 00 60 91 20 00  |.. .@. ... .`. .|
00000210  a0 9b 20 00 40 92 20 00  90 9b 20 00 70 91 20 00  |.. .@. ... .p. .|
00000220  80 9b 20 00 90 91 20 00  70 9b 20 00 c0 8b 20 00  |.. ... .p. ... .|
00000230  60 9b 20 00 a0 91 20 00  50 9b 20 00 d0 8b 20 00  |`. ... .P. ... .|
00000240  40 9b 20 00 b0 91 20 00  30 9b 20 00 a0 8b 20 00  |@. ... .0. ... .|
00000250  20 9b 20 00 b0 8b 20 00  10 9b 20 00 50 92 20 00  | . ... ... .P. .|
00000260  00 9b 20 00 30 8b 20 00  f0 9a 20 00 20 8b 20 00  |.. .0. ... . . .|
00000270  e0 9a 20 00 90 8b 20 00  d0 9a 20 00 f0 8b 20 00  |.. ... ... ... .|
00000280  c0 9a 20 00 00 8c 20 00  b0 9a 20 00 10 8c 20 00  |.. ... ... ... .|
00000290  a0 9a 20 00 90 9a 20 00  80 9a 20 00 70 9a 20 00  |.. ... ... .p. .|
000002a0  60 9a 20 00 50 9a 20 00  40 9a 20 00 e0 8b 20 00  |`. .P. .@. ... .|
000002b0  30 9a 20 00 20 9a 20 00  10 9a 20 00 00 9a 20 00  |0. . . ... ... .|
000002c0  f0 99 20 00 e0 99 20 00  d0 99 20 00 30 8c 20 00  |.. ... ... .0. .|
000002d0  c0 99 20 00 70 8c 20 00  b0 99 20 00 e0 91 20 00  |.. .p. ... ... .|
000002e0  a0 99 20 00 d0 91 20 00  90 99 20 00 10 92 20 00  |.. ... ... ... .|
000002f0  80 99 20 00 30 92 20 00  70 99 20 00 20 92 20 00  |.. .0. .p. . . .|
00000300  60 99 20 00 f0 91 20 00  50 99 20 00 00 92 20 00  |`. ... .P. ... .|
00000310  40 99 20 00 80 8b 20 00  30 99 20 00 10 8b 20 00  |@. ... .0. ... .|
00000320  20 99 20 00 70 8b 20 00  10 99 20 00 60 8b 20 00  | . .p. ... .`. .|
00000330  00 99 20 00 e0 88 20 00  f0 98 20 00 e0 98 20 00  |.. ... ... ... .|
00000340  b0 8c 20 00 d0 98 20 00  a0 8c 20 00 c0 98 20 00  |.. ... ... ... .|
00000350  e0 8a 20 00 b0 98 20 00  70 8e 20 00 a0 98 20 00  |.. ... .p. ... .|
00000360  80 8e 20 00 90 98 20 00  e0 8d 20 00 80 98 20 00  |.. ... ... ... .|
00000370  50 8b 20 00 70 98 20 00  d0 8d 20 00 60 98 20 00  |P. .p. ... .`. .|
00000380  20 89 20 00 50 98 20 00  40 89 20 00 40 98 20 00  | . .P. .@. .@. .|
00000390  30 89 20 00 30 98 20 00  50 89 20 00 20 98 20 00  |0. .0. .P. . . .|
000003a0  60 89 20 00 10 98 20 00  80 89 20 00 00 98 20 00  |`. ... ... ... .|
000003b0  90 89 20 00 f0 97 20 00  a0 8d 20 00 e0 97 20 00  |.. ... ... ... .|
000003c0  70 89 20 00 d0 97 20 00  c0 8d 20 00 c0 97 20 00  |p. ... ... ... .|
000003d0  70 8a 20 00 b0 97 20 00  b0 8d 20 00 a0 97 20 00  |p. ... ... ... .|
000003e0  60 8a 20 00 90 97 20 00  50 8a 20 00 80 97 20 00  |`. ... .P. ... .|
000003f0  40 8a 20 00 70 97 20 00  80 8a 20 00 60 97 20 00  |@. .p. ... .`. .|
00000400  a0 89 20 00 50 97 20 00  b0 89 20 00 40 97 20 00  |.. .P. ... .@. .|
00000410  30 8a 20 00 30 97 20 00  c0 89 20 00 20 97 20 00  |0. .0. ... . . .|
00000420  c0 8a 20 00 10 97 20 00  90 8a 20 00 00 97 20 00  |.. ... ... ... .|
00000430  d0 8a 20 00 f0 96 20 00  b0 8a 20 00 e0 96 20 00  |.. ... ... ... .|
00000440  a0 8a 20 00 d0 96 20 00  e0 89 20 00 c0 96 20 00  |.. ... ... ... .|
00000450  20 8a 20 00 b0 96 20 00  10 8a 20 00 a0 96 20 00  | . ... ... ... .|
00000460  00 8a 20 00 90 96 20 00  f0 89 20 00 80 96 20 00  |.. ... ... ... .|
00000470  d0 89 20 00 70 96 20 00  f0 90 20 00 60 96 20 00  |.. .p. ... .`. .|
00000480  10 91 20 00 50 96 20 00  00 91 20 00 40 96 20 00  |.. .P. ... .@. .|
00000490  30 91 20 00 30 96 20 00  20 91 20 00 20 96 20 00  |0. .0. . . . . .|
000004a0  50 91 20 00 10 96 20 00  40 91 20 00 00 96 20 00  |P. ... .@. ... .|
000004b0  f0 95 20 00 e0 95 20 00  d0 95 20 00 c0 95 20 00  |.. ... ... ... .|
000004c0  b0 95 20 00 a0 95 20 00  90 95 20 00 80 95 20 00  |.. ... ... ... .|
000004d0  70 95 20 00 60 95 20 00  00 89 20 00 50 95 20 00  |p. .`. ... .P. .|
000004e0  40 95 20 00 f0 88 20 00  30 95 20 00 20 95 20 00  |@. ... .0. . . .|
000004f0  10 95 20 00 00 95 20 00  f0 94 20 00 e0 94 20 00  |.. ... ... ... .|
00000500  d0 94 20 00 c0 94 20 00  b0 94 20 00 a0 94 20 00  |.. ... ... ... .|
00000510  10 89 20 00 90 94 20 00  70 8d 20 00 80 94 20 00  |.. ... .p. ... .|
00000520  80 8d 20 00 70 94 20 00  60 8d 20 00 60 94 20 00  |.. .p. .`. .`. .|
00000530  30 8d 20 00 50 94 20 00  40 8d 20 00 40 94 20 00  |0. .P. .@. .@. .|
00000540  50 8d 20 00 30 94 20 00  00 8d 20 00 20 94 20 00  |P. .0. ... . . .|
00000550  d0 8c 20 00 10 94 20 00  f0 8c 20 00 00 94 20 00  |.. ... ... ... .|
00000560  10 8d 20 00 f0 93 20 00  c0 8c 20 00 e0 93 20 00  |.. ... ... ... .|
00000570  e0 8c 20 00 d0 93 20 00  20 8d 20 00 c0 93 20 00  |.. ... . . ... .|
00000580  b0 93 20 00 a0 93 20 00  90 93 20 00 80 93 20 00  |.. ... ... ... .|
00000590  70 93 20 00 60 93 20 00  50 93 20 00 40 93 20 00  |p. .`. .P. .@. .|
000005a0  30 93 20 00 20 93 20 00  10 93 20 00 00 93 20 00  |0. . . ... ... .|
000005b0  f0 92 20 00 e0 92 20 00  c0 91 20 00 d0 92 20 00  |.. ... ... ... .|
000005c0  c0 92 20 00 b0 92 20 00  a0 92 20 00 90 92 20 00  |.. ... ... ... .|
000005d0  60 8c 20 00 80 92 20 00  90 8c 20 00 70 92 20 00  |`. ... ... .p. .|
000005e0  00 8b 20 00 60 92 20 00  80 8c 20 00 00 00 00 00  |.. .`. ... .....|
000005f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000008a0  e3 00 17 78 03 00 10 00  37 83 af 1c 00 00 00 00  |...x....7.......|
000008b0  e3 00 15 78 06 00 10 00  3d 83 af 1c 00 00 00 00  |...x....=.......|
000008c0  e3 00 0e 78 10 00 10 00  3f 83 af 1c 00 00 00 00  |...x....?.......|
000008d0  e3 00 04 78 04 00 10 00  39 83 af 1c 00 00 00 00  |...x....9.......|
000008e0  e1 00 80 e0 10 00 10 00  da 83 af 1c 00 00 00 00  |................|
000008f0  dc 00 2c f3 03 00 10 00  4b 84 af 1c 00 00 00 00  |..,.....K.......|
00000900  dc 00 2c f3 02 00 10 00  4a 84 af 1c 00 00 00 00  |..,.....J.......|
00000910  d8 00 6a 63 06 00 10 00  5a 84 af 1c 00 00 00 00  |..jc....Z.......|
00000920  d7 00 55 d4 0c 00 10 00  e5 83 af 1c 00 00 00 00  |..U.............|
00000930  d7 00 53 d4 0f 00 10 00  e7 83 af 1c 00 00 00 00  |..S.............|
00000940  d7 00 53 d4 0e 00 10 00  e6 83 af 1c 00 00 00 00  |..S.............|
00000950  d7 00 43 d4 14 00 10 00  e9 83 af 1c 00 00 00 00  |..C.............|
00000960  d7 00 3c d4 17 00 10 00  ea 83 af 1c 00 00 00 00  |..<.............|
00000970  d7 00 44 d4 16 00 10 00  f0 83 af 1c 00 00 00 00  |..D.............|
00000980  d7 00 47 d4 1c 00 10 00  ec 83 af 1c 00 00 00 00  |..G.............|
00000990  d7 00 47 d4 1b 00 10 00  ed 83 af 1c 00 00 00 00  |..G.............|
000009a0  d7 00 42 d4 07 00 10 00  f9 83 af 1c 00 00 00 00  |..B.............|
000009b0  d7 00 42 d4 06 00 10 00  fa 83 af 1c 00 00 00 00  |..B.............|
000009c0  d7 00 42 d4 05 00 10 00  fc 83 af 1c 00 00 00 00  |..B.............|
000009d0  d7 00 42 d4 02 00 10 00  07 84 af 1c 00 00 00 00  |..B.............|
000009e0  d7 00 41 d4 0c 00 10 00  02 84 af 1c 00 00 00 00  |..A.............|
000009f0  d7 00 41 d4 0a 00 10 00  06 84 af 1c 00 00 00 00  |..A.............|
00000a00  d7 00 3f d4 03 00 10 00  05 84 af 1c 00 00 00 00  |..?.............|
00000a10  d7 00 41 d4 06 00 10 00  04 84 af 1c 00 00 00 00  |..A.............|
00000a20  d7 00 41 d4 03 00 10 00  03 84 af 1c 00 00 00 00  |..A.............|
00000a30  d7 00 3c d4 14 00 10 00  fb 83 af 1c 00 00 00 00  |..<.............|
00000a40  d7 00 40 d4 05 00 10 00  f6 83 af 1c 00 00 00 00  |..@.............|
00000a50  d7 00 3e d4 05 00 10 00  f5 83 af 1c 00 00 00 00  |..>.............|
00000a60  d7 00 3e d4 04 00 10 00  f4 83 af 1c 00 00 00 00  |..>.............|
00000a70  d7 00 3e d4 02 00 10 00  f2 83 af 1c 00 00 00 00  |..>.............|
00000a80  d7 00 3d d4 07 00 10 00  f7 83 af 1c 00 00 00 00  |..=.............|
00000a90  d7 00 3d d4 06 00 10 00  fe 83 af 1c 00 00 00 00  |..=.............|
00000aa0  d7 00 3c d4 07 00 10 00  01 84 af 1c 00 00 00 00  |..<.............|
00000ab0  d7 00 3c d4 05 00 10 00  00 84 af 1c 00 00 00 00  |..<.............|
00000ac0  d7 00 35 d4 08 00 10 00  fd 83 af 1c 00 00 00 00  |..5.............|
00000ad0  d7 00 29 d4 1b 00 10 00  ff 83 af 1c 00 00 00 00  |..).............|
00000ae0  d1 00 76 82 08 00 10 00  dd 83 af 1c 00 00 00 00  |..v.............|
00000af0  d0 00 0f 07 03 00 10 00  42 83 af 1c 00 00 00 00  |........B.......|
00000b00  cf 00 cc 65 04 00 10 00  81 84 af 1c 00 00 00 00  |...e............|
00000b10  ce 00 4f 90 24 00 10 00  cd 83 af 1c 00 00 00 00  |..O.$...........|
00000b20  ce 00 4c 90 03 00 10 00  a3 83 af 1c 00 00 00 00  |..L.............|
00000b30  ce 00 4c 90 02 00 10 00  a2 83 af 1c 00 00 00 00  |..L.............|
00000b40  ce 00 4c 90 01 00 10 00  93 83 af 1c 00 00 00 00  |..L.............|
00000b50  c6 00 7b 9b 0e 00 10 00  e1 83 af 1c 00 00 00 00  |..{.............|
00000b60  c6 00 b7 22 34 00 10 00  d4 83 af 1c 00 00 00 00  |..."4...........|
00000b70  c6 00 b7 22 2e 00 10 00  d3 83 af 1c 00 00 00 00  |..."............|
00000b80  c6 00 b7 22 29 00 10 00  c9 83 af 1c 00 00 00 00  |...")...........|
00000b90  c6 00 b7 22 01 00 10 00  a4 83 af 1c 00 00 00 00  |..."............|
00000ba0  c6 00 b6 22 34 00 10 00  9c 83 af 1c 00 00 00 00  |..."4...........|
00000bb0  c6 00 b6 22 33 00 10 00  9e 83 af 1c 00 00 00 00  |..."3...........|
00000bc0  c6 00 b6 22 27 00 10 00  98 83 af 1c 00 00 00 00  |..."'...........|
00000bd0  c6 00 b6 22 26 00 10 00  9a 83 af 1c 00 00 00 00  |..."&...........|
00000be0  c6 00 ac 22 0c 00 10 00  af 83 af 1c 00 00 00 00  |..."............|
00000bf0  c6 00 ab 22 2f 00 10 00  a6 83 af 1c 00 00 00 00  |..."/...........|
00000c00  c6 00 ab 22 2e 00 10 00  a7 83 af 1c 00 00 00 00  |..."............|
00000c10  c6 00 ab 22 2d 00 10 00  a8 83 af 1c 00 00 00 00  |..."-...........|
00000c20  c1 00 58 58 05 00 10 00  65 83 af 1c 00 00 00 00  |..XX....e.......|
00000c30  c1 00 57 58 11 00 10 00  b5 83 af 1c 00 00 00 00  |..WX............|
00000c40  be 00 43 a3 1f 00 10 00  48 83 af 1c 00 00 00 00  |..C.....H.......|
00000c50  be 00 5a a3 0e 02 10 00  43 83 af 1c 00 00 00 00  |..Z.....C.......|
00000c60  be 00 68 5b 07 02 10 00  7f 84 af 1c 00 00 00 00  |..h[............|
00000c70  be 00 80 07 06 00 10 00  b7 83 af 1c 00 00 00 00  |................|
00000c80  b4 00 32 93 05 00 10 00  82 84 af 1c 00 00 00 00  |..2.............|
00000c90  b4 00 32 93 04 00 10 00  80 84 af 1c 00 00 00 00  |..2.............|
00000ca0  af 00 cf 71 15 00 10 00  dc 83 af 1c 00 00 00 00  |...q............|
00000cb0  af 00 cf 71 14 00 10 00  db 83 af 1c 00 00 00 00  |...q............|
00000cc0  ad 00 66 f5 0a 00 10 00  66 84 af 1c 00 00 00 00  |..f.....f.......|
00000cd0  ad 00 66 f5 07 00 10 00  63 84 af 1c 00 00 00 00  |..f.....c.......|
00000ce0  ad 00 66 f5 04 00 10 00  67 84 af 1c 00 00 00 00  |..f.....g.......|
00000cf0  ad 00 66 f5 01 00 10 00  64 84 af 1c 00 00 00 00  |..f.....d.......|
00000d00  ad 00 65 f5 1b 00 10 00  62 84 af 1c 00 00 00 00  |..e.....b.......|
00000d10  ad 00 65 f5 1a 00 10 00  65 84 af 1c 00 00 00 00  |..e.....e.......|
00000d20  ad 00 65 f5 17 00 10 00  68 84 af 1c 00 00 00 00  |..e.....h.......|
00000d30  ad 00 63 f5 1a 00 10 00  5f 84 af 1c 00 00 00 00  |..c....._.......|
00000d40  ad 00 63 f5 19 00 10 00  60 84 af 1c 00 00 00 00  |..c.....`.......|
00000d50  ad 00 63 f5 18 00 10 00  61 84 af 1c 00 00 00 00  |..c.....a.......|
00000d60  ad 00 63 f5 01 00 10 00  5e 84 af 1c 00 00 00 00  |..c.....^.......|
00000d70  ad 00 59 f5 05 00 10 00  5c 84 af 1c 00 00 00 00  |..Y.....\.......|
00000d80  ad 00 59 f5 03 00 10 00  5d 84 af 1c 00 00 00 00  |..Y.....].......|
00000d90  a9 00 fb 6f 19 00 10 00  8d 83 af 1c 00 00 00 00  |...o............|
00000da0  a3 00 63 c6 17 00 10 00  ef 83 af 1c 00 00 00 00  |..c.............|
00000db0  a3 00 63 c6 11 00 10 00  f3 83 af 1c 00 00 00 00  |..c.............|
00000dc0  a3 00 62 c6 10 00 10 00  f1 83 af 1c 00 00 00 00  |..b.............|
00000dd0  a3 00 15 02 04 00 10 00  e3 83 af 1c 00 00 00 00  |................|
00000de0  a3 00 fc 01 04 00 10 00  e0 83 af 1c 00 00 00 00  |................|
00000df0  9d 00 a2 a8 12 00 10 00  7c 83 af 1c 00 00 00 00  |........|.......|
00000e00  9d 00 a2 a8 11 00 10 00  7b 83 af 1c 00 00 00 00  |........{.......|
00000e10  9d 00 a2 a8 01 00 10 00  7d 83 af 1c 00 00 00 00  |........}.......|
00000e20  9d 00 a1 a8 16 00 10 00  7f 83 af 1c 00 00 00 00  |................|
00000e30  9d 00 a1 a8 15 00 10 00  7e 83 af 1c 00 00 00 00  |........~.......|
00000e40  9d 00 a1 a8 12 00 10 00  80 83 af 1c 00 00 00 00  |................|
00000e50  9d 00 9f a8 1e 00 10 00  7a 83 af 1c 00 00 00 00  |........z.......|
00000e60  9d 00 9f a8 15 00 10 00  78 83 af 1c 00 00 00 00  |........x.......|
00000e70  9b 00 db e8 05 00 10 00  de 83 af 1c 00 00 00 00  |................|
00000e80  9b 00 da e8 0b 00 10 00  df 83 af 1c 00 00 00 00  |................|
00000e90  99 00 ae 5e 1c 00 10 00  75 83 af 1c 00 00 00 00  |...^....u.......|
00000ea0  99 00 ae 5e 1b 00 10 00  72 83 af 1c 00 00 00 00  |...^....r.......|
00000eb0  99 00 aa 5e 0a 00 10 00  73 83 af 1c 00 00 00 00  |...^....s.......|
00000ec0  99 00 aa 5e 03 00 10 00  77 83 af 1c 00 00 00 00  |...^....w.......|
00000ed0  99 00 aa 5e 02 00 10 00  79 83 af 1c 00 00 00 00  |...^....y.......|
00000ee0  99 00 a9 5e 1b 00 10 00  76 83 af 1c 00 00 00 00  |...^....v.......|
00000ef0  99 00 a9 5e 18 00 10 00  74 83 af 1c 00 00 00 00  |...^....t.......|
00000f00  96 00 df 6d 0a 00 10 00  3a 83 af 1c 00 00 00 00  |...m....:.......|
00000f10  96 00 df 6d 06 00 10 00  56 83 af 1c 00 00 00 00  |...m....V.......|
00000f20  96 00 df 6d 04 00 10 00  3b 83 af 1c 00 00 00 00  |...m....;.......|
00000f30  96 00 df 6d 03 00 10 00  40 83 af 1c 00 00 00 00  |...m....@.......|
00000f40  96 00 df 6d 02 00 10 00  3e 83 af 1c 00 00 00 00  |...m....>.......|
00000f50  96 00 df 6d 01 00 10 00  41 83 af 1c 00 00 00 00  |...m....A.......|
00000f60  96 00 de 6d 11 00 10 00  3c 83 af 1c 00 00 00 00  |...m....<.......|
00000f70  96 00 de 6d 10 00 10 00  45 83 af 1c 00 00 00 00  |...m....E.......|
00000f80  96 00 de 6d 05 00 10 00  50 83 af 1c 00 00 00 00  |...m....P.......|
00000f90  96 00 de 6d 03 00 10 00  4f 83 af 1c 00 00 00 00  |...m....O.......|
00000fa0  96 00 de 6d 02 00 10 00  51 83 af 1c 00 00 00 00  |...m....Q.......|
00000fb0  96 00 dd 6d 0b 00 10 00  54 83 af 1c 00 00 00 00  |...m....T.......|
00000fc0  96 00 dd 6d 0a 00 10 00  53 83 af 1c 00 00 00 00  |...m....S.......|
00000fd0  96 00 dc 6d 0e 00 10 00  57 83 af 1c 00 00 00 00  |...m....W.......|
00000fe0  96 00 dc 6d 0b 00 10 00  4c 83 af 1c 00 00 00 00  |...m....L.......|
00000ff0  96 00 dc 6d 0a 00 10 00  4d 83 af 1c 00 00 00 00  |...m....M.......|
00001000  00 01 00 00 00 d7 a3 70  fd 79 12 43 41 1f 85 eb  |.......p.y.CA...|
00001010  01 1f 68 59 41 01 00 00  00 00 80 41 00 00 00 00  |..hYA......A....|
00001020  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001030  2a 00 08 00 03 29 18 0f  5d 3a 9d 6d 00 00 00 00  |*....)..]:.m....|
00001040  77 03 00 00 80 04 00 00  80 08 00 00 00 0f 00 00  |w...............|
00001050  00 13 00 00 00 1b 00 00  00 1e 00 00 00 6e 61 6d  |.............nam|
00001060  65 4b 4f 47 41 61 6d 65  6e 69 74 79 66 75 65 6c  |eKOGAamenityfuel|
00001070  66 75 65 6c 3a 6c 70 67  79 65 73 3b 00 0f 11 00  |fuel:lpgyes;....|
00001080  01 00 00 00 01 00 00 00  66 66 66 a6 ad 12 43 41  |........fff...CA|
00001090  1f 85 eb 31 ed 67 59 41  00 00 80 41 00 00 00 00  |...1.gYA...A....|
000010a0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000010b0  29 00 08 00 03 28 18 0f  5f 3a 9d 6d 00 00 00 00  |)....(.._:.m....|
000010c0  59 02 00 00 80 04 00 00  80 0a 00 00 00 12 00 00  |Y...............|
000010d0  00 17 00 00 00 6e 61 6d  65 50 52 4f 43 41 4d 6d  |.....namePROCAMm|
000010e0  61 6e 5f 6d 61 64 65 77  6f 72 6b 73 3b 00 0f 11  |an_madeworks;...|
000010f0  00 01 00 00 00 01 00 00  00 c3 f5 28 fc 8e 12 43  |...........(...C|
00001100  41 66 66 66 16 00 68 59  41 00 00 00 00 00 60 41  |Afff..hYA.....`A|
00001110  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001120  28 00 08 00 03 28 18 0f  04 e1 20 6f 00 00 00 00  |(....(.... o....|
00001130  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001140  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001150  00 01 00 00 00 cd cc cc  ec ed 1b 43 41 9a 99 99  |...........CA...|
00001160  e9 bd 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
00001170  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001180  27 00 08 00 03 28 18 0f  0b e1 20 6f 00 00 00 00  |'....(.... o....|
00001190  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000011a0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000011b0  00 01 00 00 00 7b 14 ae  e7 e5 1b 43 41 7b 14 ae  |.....{.....CA{..|
000011c0  97 be 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000011d0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000011e0  26 00 08 00 03 28 18 0f  ed e0 20 6f 00 00 00 00  |&....(.... o....|
000011f0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001200  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001210  00 01 00 00 00 9a 99 99  59 eb 1b 43 41 7b 14 ae  |........Y..CA{..|
00001220  b7 bb 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
00001230  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001240  25 00 08 00 03 28 18 0f  a7 e0 20 6f 00 00 00 00  |%....(.... o....|
00001250  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001260  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001270  00 01 00 00 00 00 00 00  a0 e8 1b 43 41 a4 70 3d  |...........CA.p=|
00001280  aa b5 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
00001290  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000012a0  24 00 08 00 03 28 18 0f  f2 e0 20 6f 00 00 00 00  |$....(.... o....|
000012b0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000012c0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000012d0  00 01 00 00 00 0a d7 a3  b0 cd 1b 43 41 29 5c 8f  |...........CA)\.|
000012e0  f2 bb 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000012f0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001300  23 00 08 00 03 28 18 0f  3b e1 20 6f 00 00 00 00  |#....(..;. o....|
00001310  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001320  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001330  00 01 00 00 00 3d 0a d7  83 b8 1b 43 41 29 5c 8f  |.....=.....CA)\.|
00001340  b2 c1 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
00001350  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001360  22 00 08 00 03 28 18 0f  3f e1 20 6f 00 00 00 00  |"....(..?. o....|
00001370  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001380  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001390  00 01 00 00 00 52 b8 1e  e5 aa 1b 43 41 8f c2 f5  |.....R.....CA...|
000013a0  d8 c1 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000013b0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000013c0  21 00 08 00 03 28 18 0f  48 e1 20 6f 00 00 00 00  |!....(..H. o....|
000013d0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000013e0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000013f0  00 01 00 00 00 a4 70 3d  8a 8d 1b 43 41 52 b8 1e  |......p=...CAR..|
00001400  35 c3 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |5.gYA......A....|
00001410  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001420  20 00 08 00 03 28 18 0f  0a e1 20 6f 00 00 00 00  | ....(.... o....|
00001430  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001440  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001450  00 01 00 00 00 29 5c 8f  c2 84 1b 43 41 d7 a3 70  |.....)\....CA..p|
00001460  5d be 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |].gYA......A....|
00001470  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001480  1f 00 08 00 03 28 18 0f  d0 df 20 6f 00 00 00 00  |.....(.... o....|
00001490  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000014a0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000014b0  00 01 00 00 00 9a 99 99  19 c3 1b 43 41 52 b8 1e  |...........CAR..|
000014c0  c5 a1 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000014d0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000014e0  1e 00 08 00 03 28 18 0f  15 df 20 6f 00 00 00 00  |.....(.... o....|
000014f0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001500  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001510  00 01 00 00 00 ae 47 e1  7a f1 1b 43 41 7b 14 ae  |......G.z..CA{..|
00001520  67 84 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |g.gYA......A....|
00001530  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001540  1d 00 08 00 03 28 18 0f  19 df 20 6f 00 00 00 00  |.....(.... o....|
00001550  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001560  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001570  00 01 00 00 00 8f c2 f5  28 da 1b 43 41 66 66 66  |........(..CAfff|
00001580  d6 84 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
00001590  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000015a0  1c 00 08 00 03 28 18 0f  92 de 20 6f 00 00 00 00  |.....(.... o....|
000015b0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000015c0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000015d0  00 01 00 00 00 48 e1 7a  b4 ee 1b 43 41 48 e1 7a  |.....H.z...CAH.z|
000015e0  54 6f 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |TogYA......A....|
000015f0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001600  1b 00 08 00 03 28 18 0f  22 df 20 6f 00 00 00 00  |.....(..". o....|
00001610  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001620  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001630  00 01 00 00 00 33 33 33  53 ca 1b 43 41 5c 8f c2  |.....333S..CA\..|
00001640  55 86 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |U.gYA......A....|
00001650  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001660  1a 00 08 00 03 28 18 0f  c3 df 20 6f 00 00 00 00  |.....(.... o....|
00001670  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001680  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001690  00 01 00 00 00 5c 8f c2  35 94 1b 43 41 ec 51 b8  |.....\..5..CA.Q.|
000016a0  de a3 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000016b0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000016c0  19 00 08 00 03 28 18 0f  40 dd 20 6f 00 00 00 00  |.....(..@. o....|
000016d0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000016e0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000016f0  00 01 00 00 00 ae 47 e1  9a 22 1b 43 41 14 ae 47  |......G..".CA..G|
00001700  a1 4b 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |.KgYA......A....|
00001710  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001720  18 00 08 00 03 28 18 0f  4e e0 20 6f 00 00 00 00  |.....(..N. o....|
00001730  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001740  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001750  00 01 00 00 00 52 b8 1e  a5 6d 1a 43 41 8f c2 f5  |.....R...m.CA...|
00001760  48 af 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |H.gYA......A....|
00001770  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001780  17 00 08 00 03 28 18 0f  12 df 20 6f 00 00 00 00  |.....(.... o....|
00001790  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000017a0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000017b0  00 01 00 00 00 e1 7a 14  4e ff 19 43 41 00 00 00  |......z.N..CA...|
000017c0  40 84 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |@.gYA......A....|
000017d0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000017e0  16 00 08 00 03 28 18 0f  0e e0 20 6f 00 00 00 00  |.....(.... o....|
000017f0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001800  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001810  00 01 00 00 00 71 3d 0a  17 d5 19 43 41 ae 47 e1  |.....q=....CA.G.|
00001820  5a a7 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |Z.gYA......A....|
00001830  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001840  15 00 08 00 03 28 18 0f  3f de 20 6f 00 00 00 00  |.....(..?. o....|
00001850  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001860  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001870  00 01 00 00 00 e1 7a 14  4e 5f 1a 43 41 b8 1e 85  |......z.N_.CA...|
00001880  9b 64 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |.dgYA......A....|
00001890  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000018a0  14 00 08 00 03 28 18 0f  5d d6 20 6f 00 00 00 00  |.....(..]. o....|
000018b0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000018c0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000018d0  00 01 00 00 00 3d 0a d7  03 31 1a 43 41 71 3d 0a  |.....=...1.CAq=.|
000018e0  d7 11 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |..gYA......A....|
000018f0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001900  13 00 08 00 03 28 18 0f  6d de 20 6f 00 00 00 00  |.....(..m. o....|
00001910  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001920  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001930  00 01 00 00 00 8f c2 f5  a8 05 1a 43 41 ae 47 e1  |...........CA.G.|
00001940  9a 6a 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |.jgYA......A....|
00001950  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001960  12 00 08 00 03 28 18 0f  2c de 20 6f 00 00 00 00  |.....(..,. o....|
00001970  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001980  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001990  00 01 00 00 00 00 00 00  e0 06 1a 43 41 c3 f5 28  |...........CA..(|
000019a0  5c 62 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |\bgYA......A....|
000019b0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
000019c0  11 00 08 00 03 28 18 0f  a4 dc 20 6f 00 00 00 00  |.....(.... o....|
000019d0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
000019e0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
000019f0  00 01 00 00 00 7b 14 ae  07 02 1a 43 41 b8 1e 85  |.....{.....CA...|
00001a00  2b 44 67 59 41 00 00 00  00 00 80 41 00 00 00 00  |+DgYA......A....|
00001a10  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001a20  10 00 08 00 03 28 18 0f  11 cd 10 6b 00 00 00 00  |.....(.....k....|
00001a30  2f 01 00 00 80 05 00 00  80 0a 00 00 00 70 6f 77  |/............pow|
00001a40  65 72 74 6f 77 65 72 3b  00 0f 11 00 01 00 00 00  |ertower;........|
00001a50  01 00 00 00 a4 70 3d ca  80 1b 43 41 a4 70 3d 0a  |.....p=...CA.p=.|
00001a60  c5 66 59 41 00 00 80 41  69 03 00 00 00 00 00 00  |.fYA...Ai.......|
00001a70  00 00 00 00 31 00 b2 22  0f 00 08 00 03 28 18 0f  |....1..".....(..|
00001a80  01 cd 10 6b 00 00 00 00  2f 01 00 00 80 05 00 00  |...k..../.......|
00001a90  80 0a 00 00 00 70 6f 77  65 72 74 6f 77 65 72 3b  |.....powertower;|
00001aa0  00 0f 11 00 01 00 00 00  01 00 00 00 c3 f5 28 bc  |..............(.|
00001ab0  da 1a 43 41 8f c2 f5 28  89 66 59 41 00 00 80 41  |..CA...(.fYA...A|
00001ac0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001ad0  0e 00 08 00 03 28 18 0f  d0 d0 20 6f 00 00 00 00  |.....(.... o....|
00001ae0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001af0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001b00  00 01 00 00 00 3d 0a d7  a3 bf 1a 43 41 00 00 00  |.....=.....CA...|
00001b10  70 ae 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |p.fYA......A....|
00001b20  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001b30  0d 00 08 00 03 28 18 0f  d2 d0 20 6f 00 00 00 00  |.....(.... o....|
00001b40  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001b50  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001b60  00 01 00 00 00 8f c2 f5  28 bc 1a 43 41 cd cc cc  |........(..CA...|
00001b70  ac b2 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001b80  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001b90  0c 00 08 00 03 28 18 0f  c5 d0 20 6f 00 00 00 00  |.....(.... o....|
00001ba0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001bb0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001bc0  00 01 00 00 00 48 e1 7a  b4 bb 1a 43 41 1f 85 eb  |.....H.z...CA...|
00001bd0  b1 ac 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001be0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001bf0  0b 00 08 00 03 28 18 0f  ca d0 20 6f 00 00 00 00  |.....(.... o....|
00001c00  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001c10  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001c20  00 01 00 00 00 52 b8 1e  45 ac 1a 43 41 e1 7a 14  |.....R..E..CA.z.|
00001c30  4e ad 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |N.fYA......A....|
00001c40  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001c50  0a 00 08 00 03 28 18 0f  9f d0 20 6f 00 00 00 00  |.....(.... o....|
00001c60  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001c70  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001c80  00 01 00 00 00 ae 47 e1  5a b0 1a 43 41 29 5c 8f  |......G.Z..CA)\.|
00001c90  02 a3 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001ca0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001cb0  09 00 08 00 03 28 18 0f  95 d0 20 6f 00 00 00 00  |.....(.... o....|
00001cc0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001cd0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001ce0  00 01 00 00 00 14 ae 47  41 eb 19 43 41 d7 a3 70  |.......GA..CA..p|
00001cf0  7d 9a 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |}.fYA......A....|
00001d00  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001d10  08 00 08 00 03 28 18 0f  8b d0 20 6f 00 00 00 00  |.....(.... o....|
00001d20  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001d30  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001d40  00 01 00 00 00 0a d7 a3  50 2c 1a 43 41 00 00 00  |........P,.CA...|
00001d50  a0 8c 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001d60  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001d70  07 00 08 00 03 28 18 0f  6b d0 20 6f 00 00 00 00  |.....(..k. o....|
00001d80  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001d90  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001da0  00 01 00 00 00 29 5c 8f  82 25 1a 43 41 1f 85 eb  |.....)\..%.CA...|
00001db0  71 80 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |q.fYA......A....|
00001dc0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001dd0  06 00 08 00 03 28 18 0f  74 d0 20 6f 00 00 00 00  |.....(..t. o....|
00001de0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001df0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001e00  00 01 00 00 00 14 ae 47  e1 21 1a 43 41 b8 1e 85  |.......G.!.CA...|
00001e10  6b 86 66 59 41 00 00 00  00 00 80 41 00 00 00 00  |k.fYA......A....|
00001e20  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001e30  05 00 08 00 03 28 18 0f  72 d0 20 6f 00 00 00 00  |.....(..r. o....|
00001e40  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001e50  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001e60  00 01 00 00 00 ae 47 e1  ba 21 1a 43 41 f6 28 5c  |......G..!.CA.(\|
00001e70  ff 83 66 59 41 01 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001e80  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001e90  04 00 08 00 03 29 18 0f  6d d0 20 6f 00 00 00 00  |.....)..m. o....|
00001ea0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001eb0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001ec0  00 01 00 00 00 e1 7a 14  2e 22 1a 43 41 a4 70 3d  |......z..".CA.p=|
00001ed0  ba 81 66 59 41 01 00 00  00 00 80 41 00 00 00 00  |..fYA......A....|
00001ee0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001ef0  03 00 08 00 03 29 18 0f  69 d0 20 6f 00 00 00 00  |.....)..i. o....|
00001f00  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001f10  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001f20  00 01 00 00 00 e1 7a 14  2e 22 1a 43 41 e1 7a 14  |......z..".CA.z.|
00001f30  ee 7e 66 59 41 01 00 00  00 00 80 41 00 00 00 00  |.~fYA......A....|
00001f40  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001f50  02 00 08 00 03 29 18 0f  5f d0 20 6f 00 00 00 00  |.....).._. o....|
00001f60  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001f70  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001f80  00 01 00 00 00 3d 0a d7  03 b7 1a 43 41 f6 28 5c  |.....=.....CA.(\|
00001f90  1f 73 66 59 41 01 00 00  00 00 80 41 00 00 00 00  |.sfYA......A....|
00001fa0  69 03 00 00 00 00 00 00  00 00 00 00 31 00 b2 22  |i...........1.."|
00001fb0  01 00 08 00 03 29 18 0f  53 d0 20 6f 00 00 00 00  |.....)..S. o....|
00001fc0  31 01 00 00 80 07 00 00  80 0b 00 00 00 6e 61 74  |1............nat|
00001fd0  75 72 61 6c 74 72 65 65  3b 00 0f 11 00 01 00 00  |uraltree;.......|
00001fe0  00 01 00 00 00 c3 f5 28  3c b3 1a 43 41 f6 28 5c  |.......(<..CA.(\|
00001ff0  0f 69 66 59 41 01 00 00  00 00 80 41 00 00 00 00  |.ifYA......A....|
00002000

-- 
Andrew (irc:RhodiumToad)

> You said a server crash occurred while you were creating indexes. Was
> that a hard crash (power failure, etc.) and is it possible that you had
> fsync disabled in postgresql.conf, or hardware or filesystems that don't
> respect fsync properly?
That was not hard crash, and fsync was disabled.
I was disconnected form remote terminal, which was performing creating 
indexes from sql file.
Next I run \di+, found missing indexes and create them manually.

> 4. Obviously the data in the corrupted block is not easily recoverable.
> If you need to recover the remaining data in the table, it could
> probably best be done by zeroing only this specific block. Whether this
> is worth doing will depend on whether it would be easier to just
> recreate the data.
If fsync was disabled when I work with table, can I restore block 
3220146 (by zeroing or maybe remove it)?

> Maybe try to read with with bt_page_items() and/or other index methods
> to confirm the theory?

bt_page_items() return: relation TABLE_NAME is not a btree index

On 4/5/18 1:46 PM, molofeev wrote:
> 
>> 4. Obviously the data in the corrupted block is not easily recoverable.
>> If you need to recover the remaining data in the table, it could
>> probably best be done by zeroing only this specific block. Whether this
>> is worth doing will depend on whether it would be easier to just
>> recreate the data.
> If fsync was disabled when I work with table, can I restore block
> 3220146 (by zeroing or maybe remove it)?

Honestly, if you've disabled fsync and it crashed, then all bets are off
and the best thing you can do is rebuilding the database. This is hardly
the only corrupted block.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Thank for help.


On 04/05/2018 03:11 PM, Tomas Vondra wrote:
> Honestly, if you've disabled fsync and it crashed, then all bets are off
> and the best thing you can do is rebuilding the database. This is hardly
> the only corrupted block.

>>>>> "molofeev" == molofeev  <molofeev3@gmail.com> writes:

 >> You said a server crash occurred while you were creating indexes. Was
 >> that a hard crash (power failure, etc.) and is it possible that you had
 >> fsync disabled in postgresql.conf, or hardware or filesystems that don't
 >> respect fsync properly?
 
 molofeev> That was not hard crash, and fsync was disabled.
 
 molofeev> I was disconnected form remote terminal, which was performing
 molofeev> creating indexes from sql file.

Just being disconnected remotely (for some reason other than a server
crash) isn't enough to account for the corruption; fsync=off only results
in corrupt data if there is an actual OS-level crash or unclean shutdown.

So I wouldn't rule out the possibility of hardware or storage subsystem
problems.

-- 
Andrew (irc:RhodiumToad)