Thread: Insert Assertion Failed in strcoll_l.c:112

Hi,

The following is written to /var/log/postgresql/postgresql-9.1-main.log:

 127.0.0.1(59800) INSERT: strcoll_l.c:112: __strcoll_l: Assertion `((uintptr_t)
indirect) % __alignof__ (indirect[0]) == 0' failed.
2012-12-23 08:54:54 SGT LOG:  server process (PID 1964) was terminated by signal
 6: Aborted
2012-12-23 08:54:54 SGT LOG:  terminating any other active server processes
2012-12-23 08:54:54 SGT WARNING:  terminating connection because of crash of ano
ther server process
2012-12-23 08:54:54 SGT DETAIL:  The postmaster has commanded this server proces
s to roll back the current transaction and exit, because another server process
exited abnormally and possibly corrupted shared memory.
2012-12-23 08:54:54 SGT HINT:  In a moment you should be able to reconnect to th
e database and repeat your command.

Error occurring on Ubuntu 12.04 with PostgreSQL 9.1 on Dell PowerEdge R710

Same code executes successfully on Ubuntu 12.10 with PostgreSQL 9.1 on Dell Precision 390. 

Application is a libpq c-application.  Application is inserting a large number of records into the DB, for later query.  Uses a thread pool and connection pool.  Connection pool establishes connections up-front and hands them to threads when required.  Connection pools creates 5 prepared statements to assist with the inserts on each connection.  70 Connections in the connection pool, up to 150 threads in the thread pool.  Mutex's manage contention on the connection pool.  Application successfully inserts 100+ records in the first table and 1000+ records in the second table before failing.  After the above error all connections begin to report 'no connection' to the database.

Configuration in /etc/postgresql/9.1/main/postgresql.conf is as follows:
fysnc =on
full_page_writes = on
max_connections = 150
max_prepared_transactions = 350
max_locks_per_transaction = 10

Configuration in /etc/sysctl.conf :
kernel.shmmax=131072000

Feels like a configuration issue.  Any suggestions on where to start looking would be appreciated.

Kind regards,
Michael

Michael Arnold <myk321@gmail.com> writes:
> The following is written to /var/log/postgresql/postgresql-9.1-main.log:

>  127.0.0.1(59800) INSERT: strcoll_l.c:112: __strcoll_l: Assertion
> `((uintptr_t)
> indirect) % __alignof__ (indirect[0]) == 0' failed.

This appears to be a failure inside glibc (specifically strcoll_l()).
There is certainly noplace in Postgres proper that would print such a
message.

> Error occurring on Ubuntu 12.04 with PostgreSQL 9.1 on Dell PowerEdge R710

> Same code executes successfully on Ubuntu 12.10 with PostgreSQL 9.1 on Dell
> Precision 390.

I would take that to mean this is a glibc bug that was fixed between
Ubuntu releases 12.04 and 12.10.  Update the older system and see if it
doesn't go away.

            regards, tom lane

Thanks Tom, but no luck this time: both boxes now on Ubuntu 12.10, but
same error.

Also removed -march gcc switch, likewise with no impact on error.

Kind regards,
Michael From: Tom Lane
Sent: 24/12/2012 1:16 AM
To: Michael Arnold
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Insert Assertion Failed in strcoll_l.c:112
Michael Arnold <myk321@gmail.com> writes:
> The following is written to /var/log/postgresql/postgresql-9.1-main.log:

>  127.0.0.1(59800) INSERT: strcoll_l.c:112: __strcoll_l: Assertion
> `((uintptr_t)
> indirect) % __alignof__ (indirect[0]) == 0' failed.

This appears to be a failure inside glibc (specifically strcoll_l()).
There is certainly noplace in Postgres proper that would print such a
message.

> Error occurring on Ubuntu 12.04 with PostgreSQL 9.1 on Dell PowerEdge R710

> Same code executes successfully on Ubuntu 12.10 with PostgreSQL 9.1 on Dell
> Precision 390.

I would take that to mean this is a glibc bug that was fixed between
Ubuntu releases 12.04 and 12.10.  Update the older system and see if it
doesn't go away.

            regards, tom lane

Michael Arnold <myk321@gmail.com> writes:
> Thanks Tom, but no luck this time: both boxes now on Ubuntu 12.10, but
> same error.

Hm.  Are they using identical locale settings?  Can you extract a
self-contained test case?

It's still a glibc bug AFAICS, but without a test case you'll not likely
get any traction reporting to the glibc guys either.

            regards, tom lane

Thanks Tom, but no luck this time: both boxes now on Ubuntu 12.10, but
same error.

Also removed -march gcc switch, likewise with no impact on error.

Kind regards,
Michael From: Tom Lane
Sent: 24/12/2012 1:16 AM
To: Michael Arnold
Cc: pgsql-general@postgresql.org
Subject: Re: [GENERAL] Insert Assertion Failed in strcoll_l.c:112
Michael Arnold <myk321@gmail.com> writes:
> The following is written to /var/log/postgresql/postgresql-9.1-main.log:

>  127.0.0.1(59800) INSERT: strcoll_l.c:112: __strcoll_l: Assertion
> `((uintptr_t)
> indirect) % __alignof__ (indirect[0]) == 0' failed.

This appears to be a failure inside glibc (specifically strcoll_l()).
There is certainly noplace in Postgres proper that would print such a
message.

> Error occurring on Ubuntu 12.04 with PostgreSQL 9.1 on Dell PowerEdge R710

> Same code executes successfully on Ubuntu 12.10 with PostgreSQL 9.1 on Dell
> Precision 390.

I would take that to mean this is a glibc bug that was fixed between
Ubuntu releases 12.04 and 12.10.  Update the older system and see if it
doesn't go away.

            regards, tom lane

Michael Arnold <myk321@gmail.com> writes:
> The locale is the same on both machines: "en_SG.UTF-8"

> I've extracted the smallest set of code which still produces the
> error.  Unfortunately its still more than 1000 lines of code, as
> without the thread and connection pools the error goes away.  If only
> 1 thread or only 1 connection is used the error also goes away.

Hm.  I ran this several times on a Fedora 16 box with locale en_SG.utf8,
and didn't see any assertion failures.  I did see a fair amount of

ERROR:  invalid byte sequence for encoding "UTF8": 0xf0 0x66 0x02
STATEMENT:  INSERT INTO TABLE1( FIELD1, FIELD2, FIELD3, FIELD4) VALUES( $1, $2, $3, $4);

with varying specific data being complained of.  Some investigation
disclosed the reason for that --- the test program is setting up the
strings to be inserted like this:

                strncpy(DataFileInfo->Field1, "AAA", 3);
                strncpy(DataFileInfo->Field2, "BBB", 3);

which of course fails to copy any trailing \0 into the struct, so unless
the struct happens to contain zeroes to start with (which is likely but
not guaranteed for a freshly-malloced block), you get strings that
appear to contain platform-specific garbage.  Changing the lengths to
"4" made the encoding complaints go away.

So at this point I surmise that the issue is triggered by unexpected
non-ASCII data, but it's going to be unlikely that anybody else can
reproduce it on the basis of this example.  Also, this bug in your
test case may or may not have much to do with the original problem...

            regards, tom lane