Thread: failed assertion in toasting code
Hello -hackers,
In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres
8.3 died at the failed assertion:
TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
LOG: server process (PID 8874) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing
LOG: database system was interrupted; last known up at 2008-02-20 07:43:00 MSK
LOG: database system was not properly shut down; automatic recovery in progress
LOG: redo starts at 78/BA00E060
LOG: record with zero length at 78/BC7A5FA8
LOG: redo done at 78/BC7A5F78
LOG: last completed transaction was at log time 2008-02-20 07:43:03.292665+03
LOG: autovacuum launcher started
LOG: database system is ready to accept connections
Unfortunately I cannot tell much more right now (I don't have
the exact name of the table even), although I suspect which column is that
(because I have only one column which is subject to toasting).
I was doing basically pg_dumpall_8.2 | psql_8.3
I can say that the tables are large ~ 100-500 Gb in size and contain the
column with the image in a special type "image", which is
toasted.
cas=# \d sdssdr5.frame Table "sdssdr5.frame" Column | Type | Modifiers
---------+------------------+----------- fieldid | bigint | not null
.................. htmid | bigint | not null img | image | not null
CREATE TYPE image ( INPUT = image_in, OUTPUT = image_out, INTERNALLENGTH = -1, STORAGE =
external
);
Although I may agree that it could be a problem occurring due to the bug
in type_in, type_out functions, I really doubt that, since it worked
perfectly with 8.1-8.2 and the code is quite simple.
I'll try to get the postmortem core dump, although it may require another
couple of days since the segfault occured after two days of dumping :(
Does anyone have ideas what could be the reason for the bug ?
Thanks in advance.
Regards, Sergey
*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru
"Sergey E. Koposov" <math@sai.msu.ru> writes:
> In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres 8.3
> died at the failed assertion:
>
> TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
...
> Does anyone have ideas what could be the reason for the bug ?
What the assert is saying is that the datum it's trying to toast is compressed
but the compressed version is larger than the original -- which shouldn't ever
happen because we don't store such data compressed.
I haven't quite figured out where the error is yet though.
-- Gregory Stark EnterpriseDB http://www.enterprisedb.com Get trained by Bruce Momjian - ask me about
EnterpriseDB'sPostgreSQL training!
> Does anyone have ideas what could be the reason for the bug ? Compression of varlena's header, introduced in 8.3. -- Teodor Sigaev E-mail: teodor@sigaev.ru WWW: http://www.sigaev.ru/
"Sergey E. Koposov" <math@sai.msu.ru> writes:
> Hello -hackers,
>
> In the process of converting a multi-Tb datadabe from 8.2 to 8.3, Postgres 8.3
> died at the failed assertion:
>
> TRAP: FailedAssertion("!(((toast_pointer).va_extsize < (toast_pointer).va_rawsize - ((int32) sizeof(int32))))", File:
"tuptoaster.c",Line: 1134)
> LOG: server process (PID 8874) was terminated by signal 6: Aborted
> LOG: terminating any other active server processes
> LOG: all server processes terminated; reinitializing
> LOG: database system was interrupted; last known up at 2008-02-20 07:43:00 MSK
> LOG: database system was not properly shut down; automatic recovery in progress
> LOG: redo starts at 78/BA00E060
> LOG: record with zero length at 78/BC7A5FA8
> LOG: redo done at 78/BC7A5F78
> LOG: last completed transaction was at log time 2008-02-20 07:43:03.292665+03
> LOG: autovacuum launcher started
> LOG: database system is ready to accept connections
>
> Unfortunately I cannot tell much more right now (I don't have the exact name of
> the table even), although I suspect which column is that (because I have only
> one column which is subject to toasting).
>
> I was doing basically pg_dumpall_8.2 | psql_8.3 I can say that the tables are
> large ~ 100-500 Gb in size and contain the column with the image in a special
> type "image", which is toasted. cas=# \d sdssdr5.frame
> Table "sdssdr5.frame"
> Column | Type | Modifiers
> ---------+------------------+-----------
> fieldid | bigint | not null
> ..................
> htmid | bigint | not null
> img | image | not null
>
> CREATE TYPE image (
> INPUT = image_in,
> OUTPUT = image_out,
> INTERNALLENGTH = -1,
> STORAGE = external
> );
You aren't doing anything funny in the image_in function to generate
compressed varlenas manually are you?
Assuming not then it must be a case where we're saving less than 4 bytes and
that's appearing as a saving in one place but then not somewhere else once you
take into account the headers. Except I've just gone through the code looking
for that kind of error and didn't spot it.
I'll keep looking (or someone else will probably spot it before I do anyways)
but if these images are mostly incompressible data you would probably be
better off marking the columns as storage "external" so Postgres just toasts
them as-is instead of trying to compress them first with:
ALTER column SET STORAGE EXTERNAL
-- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production
Tuning
"Gregory Stark" <stark@enterprisedb.com> writes: >> CREATE TYPE image ( >> INPUT = image_in, >> OUTPUT = image_out, >> INTERNALLENGTH = -1, >> STORAGE = external >> ); > > ALTER column SET STORAGE EXTERNAL Hum. I just noticed that you had set STORAGE = external in your type declaration. That makes it pretty odd that it would be going through this code path at all. Could you send select * from pg_attribute where attrelid = 'sdssdr5.frame'::regclass and attname = 'img'; -- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's On-Demand Production Tuning
On Wed, 20 Feb 2008, Gregory Stark wrote:
> You aren't doing anything funny in the image_in function to generate
> compressed varlenas manually are you?
No, I don't. The only thing I do there is unsigned char *in = PG_GETARG_CSTRING(0);//"AABBCCDDEE1122";//
and return the pointer to the palloced constructed standard varlena
datum with
typedef struct
{ int4 length; unsigned char data[1];
} image;
image *im = (image *) palloc(VARHDRSZ + out_len); memset(im, 0, VARHDRSZ + out_len); im->length =
out_len+ VARHDRSZ; /* fill the im->data ....... */ PG_RETURN_POINTER(im);
Regards, Sergey
*******************************************************************
Sergey E. Koposov
Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute
Tel: +49-6221-528-349
Web: http://lnfm1.sai.msu.ru/~math
E-mail: math@sai.msu.ru
On Wed, 20 Feb 2008, Gregory Stark wrote: > Could you send > > select * > from pg_attribute > where attrelid = 'sdssdr5.frame'::regclass > and attname = 'img'; > cas=# select * from pg_attribute where attrelid = 'sdssdr5.frame'::regclass and attname = 'img'; attrelid | attname |atttypid | attstattarget | attlen | attnum | attndims | attcacheoff | atttypmod | attbyval | attstorage | attalign | attnotnull| atthasdef | attisdropped | attislocal | attinhcount ----------+---------+----------+---------------+--------+--------+----------+-------------+-----------+----------+------------+----------+------------+-----------+--------------+------------+------------- 16856 | img | 16418 | -1 | -1 | 29 | 0 | -1 | -1 | f | e | i | t | f | f | t | 0 (1 row) Regards, Sergey ******************************************************************* Sergey E. Koposov Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute Tel: +49-6221-528-349 Web: http://lnfm1.sai.msu.ru/~math E-mail: math@sai.msu.ru
"Sergey E. Koposov" <math@sai.msu.ru> writes:
> typedef struct
> {
> int4 length;
> unsigned char data[1];
> } image;
>
> image *im = (image *) palloc(VARHDRSZ + out_len);
> memset(im, 0, VARHDRSZ + out_len);
> im->length = out_len + VARHDRSZ;
Ah, that's not going to work in 8.3 any longer. You have to change this to:
SET_VARSIZE(im, out_len+VARHDRSZ)
And you have to access the length with VARSIZE_ANY_EXHDR() (or a few other
macros but that's the most convenient).
Phew. You had me scared there.
-- Gregory Stark EnterpriseDB http://www.enterprisedb.com Ask me about EnterpriseDB's Slony Replication
support!
On Wed, 20 Feb 2008, Gregory Stark wrote: > > Ah, that's not going to work in 8.3 any longer. You have to change this to: > > SET_VARSIZE(im, out_len+VARHDRSZ) > > Phew. You had me scared there. Thank you. Sorry for scaring :) I hope that everything will work fine now. Regards, Sergey ******************************************************************* Sergey E. Koposov Max Planck Institute for Astronomy/Cambridge Institute for Astronomy/Sternberg Astronomical Institute Tel: +49-6221-528-349 Web: http://lnfm1.sai.msu.ru/~math E-mail: math@sai.msu.ru