Thread: failed assertion in toasting code

failed assertion in toasting code

From
"Sergey E. Koposov"
Date:
Hello -hackers,

In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres 
8.3 died at the failed assertion:

TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
 
LOG:  server process (PID 8874) was terminated by signal 6: Aborted
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2008-02-20 07:43:00 MSK
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 78/BA00E060
LOG:  record with zero length at 78/BC7A5FA8
LOG:  redo done at 78/BC7A5F78
LOG:  last completed transaction was at log time 2008-02-20 07:43:03.292665+03
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections

Unfortunately I cannot tell much more right now (I don't have 
the exact name of the table even), although I suspect which column is that 
(because I have only one column which is subject to toasting).

I was doing basically pg_dumpall_8.2 | psql_8.3 
I can say that the tables are large ~ 100-500 Gb in size and contain the 
column with the image in a special type "image", which is 
toasted. 
cas=# \d sdssdr5.frame         Table "sdssdr5.frame" Column  |       Type       | Modifiers
---------+------------------+----------- fieldid | bigint           | not null
.................. htmid   | bigint           | not null img     | image            | not null

CREATE TYPE image (        INPUT = image_in,        OUTPUT = image_out,        INTERNALLENGTH = -1,        STORAGE =
external
);

Although I may agree that it could be a problem occurring due to the bug 
in type_in, type_out functions, I really doubt that, since it worked 
perfectly with 8.1-8.2 and the code is quite simple.

I'll try to get the postmortem core dump, although it may require another 
couple of days since the segfault occured after two days of dumping :(

Does anyone have ideas what could be the reason for the bug ?

Thanks in advance.

Regards,    Sergey

*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru


Re: failed assertion in toasting code

From
Gregory Stark
Date:
"Sergey E. Koposov" <math@sai.msu.ru> writes:

> In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres 8.3
> died at the failed assertion:
>
> TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
 
...
> Does anyone have ideas what could be the reason for the bug ?

What the assert is saying is that the datum it's trying to toast is compressed
but the compressed version is larger than the original -- which shouldn't ever
happen because we don't store such data compressed.

I haven't quite figured out where the error is yet though.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about
EnterpriseDB'sPostgreSQL training!
 


Re: failed assertion in toasting code

From
Teodor Sigaev
Date:
> Does anyone have ideas what could be the reason for the bug ?
Compression of varlena's header, introduced in 8.3.

-- 
Teodor Sigaev                                   E-mail: teodor@sigaev.ru
  WWW: http://www.sigaev.ru/
 


Re: failed assertion in toasting code

From
Gregory Stark
Date:
"Sergey E. Koposov" <math@sai.msu.ru> writes:

> Hello -hackers,
>
> In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres 8.3
> died at the failed assertion:
>
> TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
 
> LOG:  server process (PID 8874) was terminated by signal 6: Aborted
> LOG:  terminating any other active server processes
> LOG:  all server processes terminated; reinitializing
> LOG:  database system was interrupted; last known up at 2008-02-20 07:43:00 MSK
> LOG:  database system was not properly shut down; automatic recovery in progress
> LOG:  redo starts at 78/BA00E060
> LOG:  record with zero length at 78/BC7A5FA8
> LOG:  redo done at 78/BC7A5F78
> LOG:  last completed transaction was at log time 2008-02-20 07:43:03.292665+03
> LOG:  autovacuum launcher started
> LOG:  database system is ready to accept connections
>
> Unfortunately I cannot tell much more right now (I don't have the exact name of
> the table even), although I suspect which column is that (because I have only
> one column which is subject to toasting).
>
> I was doing basically pg_dumpall_8.2 | psql_8.3 I can say that the tables are
> large ~ 100-500 Gb in size and contain the column with the image in a special
> type "image", which is toasted. cas=# \d sdssdr5.frame
>          Table "sdssdr5.frame"
>  Column  |       Type       | Modifiers
> ---------+------------------+-----------
>  fieldid | bigint           | not null
> ..................
>  htmid   | bigint           | not null
>  img     | image            | not null
>
> CREATE TYPE image (
>         INPUT = image_in,
>         OUTPUT = image_out,
>         INTERNALLENGTH = -1,
>         STORAGE = external
> );

You aren't doing anything funny in the image_in function to generate
compressed varlenas manually are you?

Assuming not then it must be a case where we're saving less than 4 bytes and
that's appearing as a saving in one place but then not somewhere else once you
take into account the headers. Except I've just gone through the code looking
for that kind of error and didn't spot it.

I'll keep looking (or someone else will probably spot it before I do anyways)
but if these images are mostly incompressible data you would probably be
better off marking the columns as storage "external" so Postgres just toasts
them as-is instead of trying to compress them first with:
ALTER column SET STORAGE EXTERNAL

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning


Re: failed assertion in toasting code

From
Gregory Stark
Date:
"Gregory Stark" <stark@enterprisedb.com> writes:


>> CREATE TYPE image (
>>         INPUT = image_in,
>>         OUTPUT = image_out,
>>         INTERNALLENGTH = -1,
>>         STORAGE = external
>> );
>
>  ALTER column SET STORAGE EXTERNAL

Hum. I just noticed that you had set STORAGE = external in your type
declaration. That makes it pretty odd that it would be going through this code
path at all.

Could you send

select *  from pg_attribute where attrelid = 'sdssdr5.frame'::regclass   and attname = 'img';

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning


Re: failed assertion in toasting code

From
"Sergey E. Koposov"
Date:
On Wed, 20 Feb 2008, Gregory Stark wrote:

> You aren't doing anything funny in the image_in function to generate
> compressed varlenas manually are you?

No, I don't. The only thing I do there is        unsigned char *in = PG_GETARG_CSTRING(0);//"AABBCCDDEE1122";//
and return the pointer to the palloced constructed standard varlena 
datum with

typedef struct
{        int4 length;        unsigned char data[1];
} image;
        image *im = (image *) palloc(VARHDRSZ + out_len);        memset(im, 0, VARHDRSZ + out_len);     im->length =
out_len+ VARHDRSZ;    /* fill the im->data    .......     */        PG_RETURN_POINTER(im);
 

Regards,    Sergey

*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru


Re: failed assertion in toasting code

From
"Sergey E. Koposov"
Date:
On Wed, 20 Feb 2008, Gregory Stark wrote:
> Could you send
>
> select *
>  from pg_attribute
> where attrelid = 'sdssdr5.frame'::regclass
>   and attname = 'img';
>

cas=# select *  from pg_attribute where attrelid = 'sdssdr5.frame'::regclass   and attname = 'img'; attrelid | attname
|atttypid | attstattarget | attlen | attnum | attndims | attcacheoff | atttypmod | attbyval | attstorage | attalign |
attnotnull| atthasdef | attisdropped | attislocal | attinhcount
 

----------+---------+----------+---------------+--------+--------+----------+-------------+-----------+----------+------------+----------+------------+-----------+--------------+------------+-------------
  16856 | img     |    16418 |            -1 |     -1 |     29 |        0 |          -1 |        -1 | f        | e
   | i        | t          | f         | f            | t          |           0
 
(1 row)

Regards,    Sergey

*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru


Re: failed assertion in toasting code

From
Gregory Stark
Date:
"Sergey E. Koposov" <math@sai.msu.ru> writes:

> typedef struct
> {
>         int4 length;
>         unsigned char data[1];
> } image;
>
>         image *im = (image *) palloc(VARHDRSZ + out_len);
>         memset(im, 0, VARHDRSZ + out_len);
>      im->length = out_len + VARHDRSZ;

Ah, that's not going to work in 8.3 any longer. You have to change this to:

SET_VARSIZE(im, out_len+VARHDRSZ)

And you have to access the length with VARSIZE_ANY_EXHDR() (or a few other
macros but that's the most convenient).

Phew. You had me scared there.

--  Gregory Stark EnterpriseDB          http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication
support!


Re: failed assertion in toasting code

From
"Sergey E. Koposov"
Date:
On Wed, 20 Feb 2008, Gregory Stark wrote:
>
> Ah, that's not going to work in 8.3 any longer. You have to change this to:
>
> SET_VARSIZE(im, out_len+VARHDRSZ)
>
> Phew. You had me scared there.

Thank you. Sorry for scaring :)
I hope that everything will work fine now.

Regards,    Sergey

*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru