Thread: Postgres error when adding new page
Hello Postgres general
So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
We're experiencing a lot of errors when using CDash on PostgreSQL 9.1, hosted on Mac OSX 10.6.8. The actual error message is as follows:
SQL error in Cannot insert test: utility/asserter/assert_file_returns_true_for_empty_files into the database():ERROR: failed to add old item to the right sibling while splitting block 191 of index "crc323"<br>
After some investigation, it appears the error is coming from the guts of Postgres: src/backend/access/nbtree/nbtinsert.c:1077:
if (!_bt_pgaddtup(rightpage, itemsz, item, rightoff))
{
memset(rightpage, 0, BufferGetPageSize(rbuf));
elog(ERROR, "failed to add old item to the right sibling"
" while splitting block %u of index \"%s\"",
origpagenumber, RelationGetRelationName(rel));
}
rightoff = OffsetNumberNext(rightoff);
I'm a bit stuck from here on. Is the likely reason for this problem filesystem corruption or am I barking at the wrong tree?
Many thanks for your time.
Marco
-- So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
On 1 October 2012 14:47, Marco Craveiro <marco.craveiro@gmail.com> wrote: > Hello Postgres general > > We're experiencing a lot of errors when using CDash on PostgreSQL 9.1, > hosted on Mac OSX 10.6.8. The actual error message is as follows: > > SQL error in Cannot insert test: > utility/asserter/assert_file_returns_true_for_empty_files into the > database():ERROR: failed to add old item to the right sibling while > splitting block 191 of index "crc323"<br> A call to PageAddItem(), made within _bt_pgaddtup(), is where this failure seems to ultimately originate from. What we're missing here is the reason for PageAddItem() returning InvalidOffsetNumber. That is usually, though not necessarily, separately available within a WARNING log message, which you haven't included here. Could you please let us know if there is a WARNING that you didn't include just prior to the ERROR? -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services
Peter,
Thanks for your prompt reply.
--
So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
A call to PageAddItem(), made within _bt_pgaddtup(), is where this
failure seems to ultimately originate from. What we're missing here is
the reason for PageAddItem() returning InvalidOffsetNumber. That is
usually, though not necessarily, separately available within a WARNING
log message, which you haven't included here. Could you please let us
know if there is a WARNING that you didn't include just prior to the
ERROR?
No warning I'm afraid. These are the statements I see on the Postgres log file:
2012-10-01 13:09:12 WEST ERROR: failed to add old item to the right sibling while splitting block 191 of index "crc323"
2012-10-01 13:09:12 WEST STATEMENT: INSERT INTO test (projectid,crc32,name,path,command,details,output)
VALUES ('2','2548249718','utility/xml/closing_an_open_text_reader_does_not_throw','./projects/utility/spec','e:\cmake\bin\cmake.exe "-E" "chdir" "E:/mingw/msys/1.0/home/ctest/build/Continuous/dogen/mingw-1.0.17-i686-gcc-4.7/build/stage/bin" "E:/mingw/msys/1.0/home/ctest/build/Continuous/dogen/mingw-1.0.17-i686-gcc-4.7/build/stage/bin/dogen_utility_spec" "--run_test=xml/closing_an_open_text_reader_does_not_throw"','Completed','UnVubmluZyAxIHRlc3QgY2FzZS4uLgoKKioqIE5vIGVycm9ycyBkZXRlY3RlZAo=')
These are repeated several times as CDash keeps on retrying. After a few retries we succeed (the actual number of retries is variable - 8, 10, etc).
Cheers
Marco
So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
Marco Craveiro <marco.craveiro@gmail.com> writes:
> We're experiencing a lot of errors when using CDash on PostgreSQL 9.1,
> hosted on Mac OSX 10.6.8. The actual error message is as follows:
> SQL error in Cannot insert test:
> utility/asserter/assert_file_returns_true_for_empty_files into the
> database():ERROR: failed to add old item to the right sibling while
> splitting block 191 of index "crc323"<br>
> I'm a bit stuck from here on. Is the likely reason for this problem
> filesystem corruption or am I barking at the wrong tree?
This definitely looks like index corruption, but blaming it on the
filesystem might be premature. I'm wondering if this could be an
artifact of the WAL-replay bug fixed in 9.1.6. I'd suggest updating
and then reindexing the index ...
regards, tom lane
Hello Tom,
So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
This definitely looks like index corruption, but blaming it on the
filesystem might be premature. I'm wondering if this could be an
artifact of the WAL-replay bug fixed in 9.1.6. I'd suggest updating
and then reindexing the index ...
We are running 9.1.2 it seems:
select version();
version
----------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 9.1.2 on x86_64-apple-darwin, compiled by i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3), 64-bit
We'll look into upgrading to 9.1.6.
Cheers
Marco
-- So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com
Tom, Peter,
I'm wondering if this could be an artifact of the WAL-replay bug fixed in 9.1.6. I'd suggest updating and then reindexing the index ...We are running 9.1.2 it seems
We did a file system check and it all appeared green, at least as far as OSX is concerned. We then upgraded, but to save us time we moved directly to 9.2.1. This is not ideal in terms of root-causing the underlying problem but as we can't afford the time to do multiple upgrades, we had to go for it.
CDash is now very happy - no errors on the log. Many thanks for your help.
Cheers
Marco
-- So young, and already so unknown -- Pauli
blog: http://mcraveiro.blogspot.com