Thread: PostgreSQL 7.3.3 with pgcrypto on FreeBSD 5.1
============================================================================
POSTGRESQL BUG REPORT TEMPLATE
============================================================================
Your name : Alexey Rodin
Your email address : alx@sm.ukrtel.net
System Configuration
---------------------
Architecture : CPU: AMD Duron(tm) processor (801.42-MHz 686-class CPU)
Operating System : FreeBSD 5.1-RELEASE
PostgreSQL version : PostgreSQL-7.3.3
Compiler used : gcc version 3.2.2 [FreeBSD] 20030205 (release)
Please enter a FULL description of your problem:
------------------------------------------------
PostgreSQL 7.3.3 with pgcrypto crushed after execute test sql query.
Please describe a way to repeat the problem. Please try to provide a
concise reproducible example, if at all possible:
----------------------------------------------------------------------
testdb=# SELECT crypt('lalalal',gen_salt('md5'));
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
<pglog>
LOG: ReadRecord: record with zero length at 0/18BDEA8
LOG: redo is not required
LOG: database system is ready
LOG: server process (pid 21460) was terminated by signal 11
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing shared memory and semaphores
LOG: database system was interrupted at 2003-07-21 11:44:47 EEST
LOG: checkpoint record is at 0/18BDEA8
LOG: redo record is at 0/18BDEA8; undo record is at 0/0; shutdown TRUE
LOG: next transaction id: 996; next oid: 98896
LOG: database system was not properly shut down; automatic recovery in progress
FATAL: The database system is starting up
</pglog>
If you know how this problem might be fixed, list the solution below:
---------------------------------------------------------------------
> Please describe a way to repeat the problem. Please try to provide a
> concise reproducible example, if at all possible:
> ----------------------------------------------------------------------
>
> testdb=# SELECT crypt('lalalal',gen_salt('md5'));
>
> server closed the connection unexpectedly
> This probably means the server terminated abnormally
> before or while processing the request.
> The connection to the server was lost. Attempting reset: Failed.
>
> <pglog>
>
> LOG: ReadRecord: record with zero length at 0/18BDEA8
> LOG: redo is not required
> LOG: database system is ready
> LOG: server process (pid 21460) was terminated by signal 11
> LOG: terminating any other active server processes
> LOG: all server processes terminated; reinitializing shared memory and semaphores
> LOG: database system was interrupted at 2003-07-21 11:44:47 EEST
> LOG: checkpoint record is at 0/18BDEA8
> LOG: redo record is at 0/18BDEA8; undo record is at 0/0; shutdown TRUE
> LOG: next transaction id: 996; next oid: 98896
> LOG: database system was not properly shut down; automatic recovery in progress
> FATAL: The database system is starting up
> </pglog>
FWIW, I can confirm this, but I don't think it's a FreeBSD specific
problem given that the backend dies inside of an OpenSSL routine.
#0 0x2864ae9c in EVP_DigestUpdate () from /usr/lib/libcrypto.so.3
#1 0x28576a90 in px_find_cipher () from /usr/local/lib/postgresql/pgcrypto.so
#2 0x2857c584 in px_crypt_md5 () from /usr/local/lib/postgresql/pgcrypto.so
#3 0x2857efa4 in px_gen_salt () from /usr/local/lib/postgresql/pgcrypto.so
#4 0x2857ee1f in px_crypt () from /usr/local/lib/postgresql/pgcrypto.so
#5 0x2857d2be in pg_crypt () from /usr/local/lib/postgresql/pgcrypto.so
#6 0x0811acb6 in ExecMakeFunctionResult ()
#7 0x0811b7db in ExecEvalExpr ()
#8 0x0811e794 in ExecProject ()
#9 0x0811cb7a in ExecProject ()
#10 0x081282a5 in ExecResult ()
#11 0x0811a0e5 in ExecProcNode ()
#12 0x0811952c in ExecEndPlan ()
#13 0x081188b8 in ExecutorRun ()
#14 0x0819b703 in PortalRun ()
#15 0x0819b53e in PortalRun ()
#16 0x08199b21 in authdie ()
#17 0x081982fd in PostgresMain ()
#18 0x0816de64 in PostmasterMain ()
#19 0x0816d628 in PostmasterMain ()
#20 0x0816bcc6 in PostmasterMain ()
#21 0x0816b5c7 in PostmasterMain ()
#22 0x0813588b in main ()
#23 0x0806c802 in _start ()
Both crypt() and gen_salt() work independently of each other:
test=# SELECT gen_salt('md5');
gen_salt
-------------
$1$nouzuI/B
(1 row)
test=# SELECT crypt('foo','ba');
crypt
---------------
ba4TuD1iozTxw
(1 row)
:-/ crypt() does die with the salt, '$1$' Not sure why either...
That said, the right work around is to _not_ use gen_salt('md5') when
passing data in to crypt(), instead, use gen_salt('des'). The readme
says its not recommended, but in the same breath, it should recommend
that gen_salt('des') is as weak as crypt() and gen_salt('des')
produces the correct salt for use with crypt().
What's interesting, however, is that I can no longer ctrl+c to get out
of psql after the backend dies. To kill psql, I have to suspend it,
then kill.
test=# SELECT crypt('lalal',gen_salt('md5'));
The connection to the server was lost. Attempting reset: Failed.
connection pointer is NULL
!>
!>
!>
!>
!>
!>
!>
[1] + 15252 Suspended psql test pgsql
1:08pm sean@localhost:databases/postgresql-devel > kill %1
1:08pm sean@localhost:databases/postgresql-devel >
[1] Terminated psql test pgsql
-sc
--
Sean Chittenden
Sean Chittenden <sean@chittenden.org> writes:
>> testdb=# SELECT crypt('lalalal',gen_salt('md5'));
>> server closed the connection unexpectedly
> FWIW, I can confirm this, but I don't think it's a FreeBSD specific
> problem given that the backend dies inside of an OpenSSL routine.
Works fine here:
regression=# SELECT crypt('lalalal',gen_salt('md5'));
crypt
------------------------------------
$1$2tnDkstz$e2vTadSh2zGH4yh51Ocbu0
(1 row)
"here" being Red Hat 8.0 with these OpenSSL packages:
openssl095a-0.9.5a-16
openssl-devel-0.9.6b-29
openssl-perl-0.9.6b-29
openssl-0.9.6b-29
openssl096-0.9.6-11
What OpenSSL release are you using?
regards, tom lane
Sean Chittenden <sean@chittenden.org> writes:
> FWIW, I can confirm this, but I don't think it's a FreeBSD specific
> problem given that the backend dies inside of an OpenSSL routine.
> #0 0x2864ae9c in EVP_DigestUpdate () from /usr/lib/libcrypto.so.3
> #1 0x28576a90 in px_find_cipher () from /usr/local/lib/postgresql/pgcrypto.so
> #2 0x2857c584 in px_crypt_md5 () from /usr/local/lib/postgresql/pgcrypto.so
By chance I noticed this entry in the CVS logs:
2002-11-14 21:54 momjian
* contrib/pgcrypto/openssl.c: OpenSSL 0.9.6g in Debian/unstable
stopped working with pgcrypto. This is pgcrypto bug as it assumed
too much about inner workings of OpenSSL.
Following patch stops pgcrypto using EVP* functions for ciphers and
lets it manage ciphers itself.
This patch supports Blowfish, DES and CAST5 algorithms.
Marko Kreen
Perhaps the problem is that Marko didn't fix the crypt() code in the
same way?
regards, tom lane
> > FWIW, I can confirm this, but I don't think it's a FreeBSD > > specific problem given that the backend dies inside of an OpenSSL > > routine. > > > #0 0x2864ae9c in EVP_DigestUpdate () from /usr/lib/libcrypto.so.3 > > #1 0x28576a90 in px_find_cipher () from /usr/local/lib/postgresql/pgcrypto.so > > #2 0x2857c584 in px_crypt_md5 () from /usr/local/lib/postgresql/pgcrypto.so > > By chance I noticed this entry in the CVS logs: > > 2002-11-14 21:54 momjian > > * contrib/pgcrypto/openssl.c: OpenSSL 0.9.6g in Debian/unstable > stopped working with pgcrypto. This is pgcrypto bug as it assumed > too much about inner workings of OpenSSL. > > Following patch stops pgcrypto using EVP* functions for ciphers and > lets it manage ciphers itself. > > This patch supports Blowfish, DES and CAST5 algorithms. > > Marko Kreen > > Perhaps the problem is that Marko didn't fix the crypt() code in the > same way? Ah, I think that's _very_ likely the case here... -sc -- Sean Chittenden
> >> testdb=# SELECT crypt('lalalal',gen_salt('md5'));
> >> server closed the connection unexpectedly
>
> > FWIW, I can confirm this, but I don't think it's a FreeBSD specific
> > problem given that the backend dies inside of an OpenSSL routine.
>
> Works fine here:
>
> regression=# SELECT crypt('lalalal',gen_salt('md5'));
> crypt
> ------------------------------------
> $1$2tnDkstz$e2vTadSh2zGH4yh51Ocbu0
> (1 row)
>
> "here" being Red Hat 8.0 with these OpenSSL packages:
>
> openssl095a-0.9.5a-16
> openssl-devel-0.9.6b-29
> openssl-perl-0.9.6b-29
> openssl-0.9.6b-29
> openssl096-0.9.6-11
>
> What OpenSSL release are you using?
0.9.7a, which lends weight to the theory of shifting internals...
-sc
--
Sean Chittenden
Sean Chittenden <sean@chittenden.org> writes:
>> Perhaps the problem is that Marko didn't fix the crypt() code in the
>> same way?
> Ah, I think that's _very_ likely the case here... -sc
I updated to openssl 0.9.7b on my HPUX machine, and still do not see any
failure in
regression=# SELECT crypt('lalalal',gen_salt('md5'));
crypt
------------------------------------
$1$.yzzlyzz$W8vpUQ3Nonx20vchlBQye/
(1 row)
So it seems that OpenSSL version is not the issue --- or at least not
the whole story. Maybe there is some platform-dependency here too.
regards, tom lane
> >> Perhaps the problem is that Marko didn't fix the crypt() code in
> >> the same way?
>
> > Ah, I think that's _very_ likely the case here... -sc
>
> I updated to openssl 0.9.7b on my HPUX machine, and still do not see any
> failure in
>
> regression=# SELECT crypt('lalalal',gen_salt('md5'));
> crypt
> ------------------------------------
> $1$.yzzlyzz$W8vpUQ3Nonx20vchlBQye/
> (1 row)
>
> So it seems that OpenSSL version is not the issue --- or at least not
> the whole story. Maybe there is some platform-dependency here too.
Possible, but I'm a skeptical of that. FreeBSD's openssl code
_should_ be stock (unless someone bungled the import) with the
exception of not including Win32 or other non-FreeBSD related bits.
crypt() works when salted with only 2 chars, however it shouldn't core
with more than that... I ran this test case through gdb a few times
and couldn't come up with anything worth while, though I'm pretty sure
that's because the object files were optimized > -O and garbled.
-sc
--
Sean Chittenden
Sean Chittenden <sean@chittenden.org> writes:
> Possible, but I'm a skeptical of that. FreeBSD's openssl code
> _should_ be stock (unless someone bungled the import) with the
> exception of not including Win32 or other non-FreeBSD related bits.
> crypt() works when salted with only 2 chars, however it shouldn't core
> with more than that...
But maybe the problem is in crypt(3) and not in OpenSSL at all? I'm
quite willing to believe that neither Linux nor HPUX share crypt()
source with FreeBSD...
regards, tom lane
> > Possible, but I'm a skeptical of that. FreeBSD's openssl code > > _should_ be stock (unless someone bungled the import) with the > > exception of not including Win32 or other non-FreeBSD related > > bits. crypt() works when salted with only 2 chars, however it > > shouldn't core with more than that... > > But maybe the problem is in crypt(3) and not in OpenSSL at all? I'm > quite willing to believe that neither Linux nor HPUX share crypt() > source with FreeBSD... Using the builtin crypt() solved this problem for the user though. I wonder if the configure script isn't correctly detecting using the builtin crypt() as opposed to the openssl version... or visa versa. Try applying the attached patch and seeing if that lets you reproduce the crash. random may still need to be silly on HPUX, I don't know if it has a /dev/random. -sc -- Sean Chittenden
Attachment
Sean Chittenden <sean@chittenden.org> writes:
> Try applying the attached patch and seeing if that lets you reproduce
> the crash.
Hmm. I get *a* crash, maybe not the same one. But there's no
autoconfiguration of this setting in pgcrypto/Makefile, so how would
anyone be using anything but builtin crypt()?
regards, tom lane
> > Try applying the attached patch and seeing if that lets you > > reproduce the crash. > > Hmm. I get *a* crash, maybe not the same one. But there's no > autoconfiguration of this setting in pgcrypto/Makefile, so how would > anyone be using anything but builtin crypt()? I patch contrib/pgcrypto/Makefile in FreeBSD's postgresql-devel port to turn on the use of the OpenSSL where applicable. FreeBSD and OpenBSD have hardware offloading support for crypto routines, but offloading (the speedup of crypto handling is measured in orders of magnitude) only takes place when OpenSSL's lib handle the crypto, hence the patch. I'm torn as to what to do, I'll probably conditionalize that part of the patch behind an #ifdef unless a patch comes through shortly. My bet is Pg is smashing OpenSSL's stack when passing more than 2 chars as a salt. When passed the right arguments, pgcrypto works correctly, but it falls on its face when OpenSSL is handed out of bounds data. More than anything, I think I should submit a quick doc patch to the README so that people use a DES salt when using a DES based crypt() routine and an md5 salt when calling an md5 crypt routine, which is where the bug submitter triggered this boundary condition. -sc -- Sean Chittenden
Sean Chittenden <sean@chittenden.org> writes:
> My bet is Pg is smashing OpenSSL's stack when
> passing more than 2 chars as a salt.
I looked at the code a little bit and that doesn't seem to be the case.
On my machine the core dump seems to be because EVP_DigestUpdate is
called with a EVP_MD_CTX that's all zeroes. So I think the previous
theory about not having tracked an API change is on target after all.
Personally, I'm punting this problem in Marko's direction --- I won't
take responsibility for fixing contrib code that isn't even compiled
by default.
regards, tom lane