Thread: panic failed to add item
I get this running mnogosearch against my 8.4.1 database consistently:
PANIC: failed to add item to the right sibling in index "logged_in_uid"
STATEMENT: INSERT INTO logged_in (orgid, uid, remote_addr,
orig_session_id, new_session_id) VALUES ('394746', '1125200',
'24.251.180.193', 'c2f5bfe61be2ea80548c46c156f2242d',
'2e777119d2bf0d5e35b653b80f3804e8');
LOG: server process (PID 27524) was terminated by signal 6: Aborted
LOG: terminating any other active server processes
When I run against an 8.3.8 database I do not. It takes days to see
this error and it scrams whatever indexer is running at the time. for
now I don't have time to do a lot of testing, but if anyone wants me
to try anything to track it down or something I'm all ears. And
possibly left thumbs.
Scott Marlowe <scott.marlowe@gmail.com> writes:
> I get this running mnogosearch against my 8.4.1 database consistently:
> PANIC: failed to add item to the right sibling in index "logged_in_uid"
Huh. Don't suppose you can extract a reproducible test case ;-).
What are the exact definitions of the table and index?
regards, tom lane
On Fri, Nov 20, 2009 at 2:33 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Scott Marlowe <scott.marlowe@gmail.com> writes:
>> I get this running mnogosearch against my 8.4.1 database consistently:
>> PANIC: failed to add item to the right sibling in index "logged_in_uid"
>
> Huh. Don't suppose you can extract a reproducible test case ;-).
> What are the exact definitions of the table and index?
Oh hey, whaddya know, the problem was a symptom that showed up in
mnogosearch, but it came from our stats database.
\d logged_in
Table "public.logged_in"
Column | Type |
Modifiers
-----------------+--------------------------+------------------------------------------------------------------
logged_in_id | integer | not null default
nextval('logged_in_logged_in_id_seq'::regclass)
uid | integer |
timestamp | timestamp with time zone | default now()
remote_addr | text |
orgid | integer |
orig_session_id | text |
new_session_id | text |
logout_reason | text |
Indexes:
"logged_in_id" PRIMARY KEY, btree (logged_in_id)
"logged_in_timestamp" btree ("timestamp")
"logged_in_uid" btree (uid)
Problem is we insert 125k rows a day and get that signal 6 every one
to four days. So a reproduceable test case may involved insertinto
400k or more rows to make it happen. I'll see what I can do.
Scott Marlowe <scott.marlowe@gmail.com> writes:
>>> I get this running mnogosearch against my 8.4.1 database consistently:
>>> PANIC: �failed to add item to the right sibling in index "logged_in_uid"
Hmm, if uid is an integer then all the index entries will be the same
size, which eliminates my first theory about there being some corner
case where we're (still) choosing an infeasible split point. Can you
comment about the insertion pattern for uid's --- are they ever-increasing
for instance? Do you have any special properties such as fillfactor set
on that index?
regards, tom lane
On Fri, Nov 20, 2009 at 3:15 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Scott Marlowe <scott.marlowe@gmail.com> writes: >>>> I get this running mnogosearch against my 8.4.1 database consistently: >>>> PANIC: failed to add item to the right sibling in index "logged_in_uid" > > Hmm, if uid is an integer then all the index entries will be the same > size, which eliminates my first theory about there being some corner > case where we're (still) choosing an infeasible split point. Can you > comment about the insertion pattern for uid's --- are they ever-increasing > for instance? Do you have any special properties such as fillfactor set > on that index? Access is pretty random actually, and fill factor on this database is 100% because it doesn't really get updated, just appended to. There are a lot of parallel insertions going on if that helps.
Scott Marlowe <scott.marlowe@gmail.com> writes:
> Access is pretty random actually, and fill factor on this database is
> 100% because it doesn't really get updated, just appended to. There
> are a lot of parallel insertions going on if that helps.
Do you mean you actually have fillfactor set somewhere? It didn't show
on the \d output.
Is this a 32-bit or 64-bit machine?
regards, tom lane
On Fri, Nov 20, 2009 at 4:03 PM, Tom Lane <tgl@sss.pgh.pa.us> wrote: > Scott Marlowe <scott.marlowe@gmail.com> writes: >> Access is pretty random actually, and fill factor on this database is >> 100% because it doesn't really get updated, just appended to. There >> are a lot of parallel insertions going on if that helps. > > Do you mean you actually have fillfactor set somewhere? It didn't show > on the \d output. No, I just assumed it was 100% still by default. Did that change? > Is this a 32-bit or 64-bit machine? 64 bit. Centos / RHEL 5.3 running the -95 or so Centos 5.2 kernel because later kernels cause problems with the Areca 1680 series RAID array controller kicking offline at odd times