Thread: Segmentation fault using digest from pg_crypto
Hi, I have a simple query that uses digest(data, 'sha1') and consistently crashes the backend (tested in two different machines): ciencias=# select version(); version ---------------------------------------------------------------------------------------------------------PostgreSQL 8.2.4on x86_64-redhat-linux-gnu, compiled by GCC gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-8) (1 row) ciencias=# select version(); version --------------------------------------------------------------------------------------------------------PostgreSQL 8.2.4on i686-redhat-linux-gnu, compiled by GCC gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-51) (1 fila) with segmentation fault. Backtrace follows: -bash-3.1$ gdb `which postgres` ./data/core.7428 GNU gdb Red Hat Linux (6.5-15.fc6rh) Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". warning: Can't read pathname for load map: Error de entrada/salida. Reading symbols from /lib/libcrypt.so.1...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libdl.so.2...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /usr/lib/gconv/ISO8859-1.so...done. Loaded symbols for /usr/lib/gconv/ISO8859-1.so Reading symbols from /usr/lib/pgsql/pgcrypto.so...Reading symbols from /usr/lib/debug/usr/lib/pgsql/pgcrypto.so.debug...done. done. Loaded symbols for /usr/lib/pgsql/pgcrypto.so Reading symbols from /usr/lib/libz.so.1...done. Loaded symbols for /usr/lib/libz.so.1 Core was generated by `postgres: masm ciencias [local] SELECT '. Program terminated with signal 11, Segmentation fault. #0 pg_detoast_datum (datum=0x0) at fmgr.c:1964 1964 if (VARATT_IS_EXTENDED(datum)) (gdb) bt #0 pg_detoast_datum (datum=0x0) at fmgr.c:1964 #1 0x00124c9e in pg_digest (fcinfo=0xbf9ba174) at pgcrypto.c:73 #2 0x0815127c in ExecMakeFunctionResultNoSets (fcache=0x8fa84f8, econtext=0x8fa83d8, isNull=0x8fad282 "", isDone=0x8fad2e0)at execQual.c:1330 #3 0x0814ce57 in ExecProject (projInfo=0x8fad290, isDone=0xbf9ba438) at execQual.c:4119 #4 0x08159d1c in ExecHashJoin (node=0x8fa8350) at nodeHashjoin.c:270 #5 0x0814c55d in ExecProcNode (node=0x8fa8350) at execProcnode.c:390 #6 0x0814b87b in ExecutorRun (queryDesc=0x8fa7ae0, direction=ForwardScanDirection, count=0) at execMain.c:1082 #7 0x081d5300 in PortalRunSelect (portal=0x8f85530, forward=<value optimized out>, count=0, dest=0x8f97920) at pquery.c:831 #8 0x081d6269 in PortalRun (portal=0x8f85530, count=2147483647, dest=0x8f97920, altdest=0x8f97920, completionTag=0xbf9ba688"") at pquery.c:684 #9 0x081d1f40 in exec_simple_query ( query_string=0x8f64420 "select a.id, a.persona, digest(to_char(fecha_nacimiento,'DDMMYYYY'), 'sha1') from auth.credential c join auth.agent a on (a.id = c.agent) join persona.personap on (p.id = a.persona) where now() betw"...) at postgres.c:939 #10 0x081d3517 in PostgresMain (argc=4, argv=<value optimized out>, username=0x8efc540 "masm") at postgres.c:3424 #11 0x081aaf0e in ServerLoop () at postmaster.c:2931 #12 0x081abd0a in PostmasterMain (argc=3, argv=0x8ef95b8) at postmaster.c:963 #13 0x0816b3d0 in main (argc=3, argv=Cannot access memory at address 0x9 ) at main.c:188 (gdb) Please letme know if there is more information I can supply. Regards, Manuel.
Manuel Sugawara <masm@fciencias.unam.mx> writes: > Hi, > > I have a simple query that uses digest(data, 'sha1') and consistently > crashes the backend The problem is with null arguments: ciencias=# select digest(null, 'sha1'); el servidor ha cerrado la conexión inesperadamente, probablemente porque terminó de manera anormal antes o duranteel procesamiento de la petición. La conexión al servidor se ha perdido. Intentando reiniciar: falló. !> Regards, Manuel.
On 8/24/07, Manuel Sugawara <masm@fciencias.unam.mx> wrote: > Manuel Sugawara <masm@fciencias.unam.mx> writes: > > I have a simple query that uses digest(data, 'sha1') and consistently > > crashes the backend > > The problem is with null arguments: > > ciencias=# select digest(null, 'sha1'); In 8.0 the pgcrypto functions were non-strict and checked for NULLs. In 8.1 they were made STRICT. In 8.2 the NULL check were removed from code. That means if you use function signatures from 8.0 with pgcrypto from 8.2, they will crash on NULL. To fix it, just reinitialize functions: \i pgcrypto.sql -- marko
Manuel Sugawara wrote: >> I have a simple query that uses digest(data, 'sha1') and consistently >> crashes the backend > > The problem is with null arguments: > > ciencias=# select digest(null, 'sha1'); > el servidor ha cerrado la conexión inesperadamente, Can you check if the function is defined "STRICT"? That should automatically return null if a parameter is null without calling the function. Simplest way to check is probably to pg_dump --schema-only and search for the function defn. -- Richard Huxton Archonet Ltd
"Marko Kreen" <markokr@gmail.com> writes: > On 8/24/07, Manuel Sugawara <masm@fciencias.unam.mx> wrote: >> Manuel Sugawara <masm@fciencias.unam.mx> writes: >> > I have a simple query that uses digest(data, 'sha1') and consistently >> > crashes the backend >> >> The problem is with null arguments: >> >> ciencias=# select digest(null, 'sha1'); > > In 8.0 the pgcrypto functions were non-strict and checked for NULLs. > > In 8.1 they were made STRICT. > > In 8.2 the NULL check were removed from code. Not an smart move IMHO, I didn't create the function, it was created as part of my upgrade process. May I suggest to put back the check?. Regards, Manuel.
Manuel Sugawara <masm@fciencias.unam.mx> writes:
> "Marko Kreen" <markokr@gmail.com> writes:
>> In 8.0 the pgcrypto functions were non-strict and checked for NULLs.
>> In 8.1 they were made STRICT.
>> In 8.2 the NULL check were removed from code.
> Not an smart move IMHO, I didn't create the function, it was created
> as part of my upgrade process. May I suggest to put back the check?.
That's the standard way of doing things in C functions (ie, rely on
STRICT markings) and I see nothing wrong with it. If you were using an
upgrade process that failed to incorporate version-to-version changes in
the definitions of contrib functions, this is hardly going to be the
only problem you encounter.
regards, tom lane
Tom Lane <tgl@sss.pgh.pa.us> writes: > Manuel Sugawara <masm@fciencias.unam.mx> writes: >> "Marko Kreen" <markokr@gmail.com> writes: >>> In 8.0 the pgcrypto functions were non-strict and checked for NULLs. >>> In 8.1 they were made STRICT. >>> In 8.2 the NULL check were removed from code. > >> Not an smart move IMHO, I didn't create the function, it was created >> as part of my upgrade process. May I suggest to put back the check?. > > That's the standard way of doing things in C functions (ie, rely on > STRICT markings) and I see nothing wrong with it. > > If you were using an upgrade process that failed to incorporate > version-to-version changes in the definitions of contrib functions, > this is hardly going to be the only problem you encounter. I was under the impression that the standar procedure for upgrading was to read the release notes searching for this kind of gotchas. I've reviewed (again) the release notes for 8.1 and 8.2 and didn't find anything related but maybe I'm not searching hard enough. If something is not going a work (or is going a work in a different way) in some version after loading a shot form a previous one I think it should be documented or some kind of backwards compatibility mechanism should be provided. Regards, Manuel.
On 8/24/07, Manuel Sugawara <masm@fciencias.unam.mx> wrote: > Tom Lane <tgl@sss.pgh.pa.us> writes: > > Manuel Sugawara <masm@fciencias.unam.mx> writes: > >> "Marko Kreen" <markokr@gmail.com> writes: > >>> In 8.0 the pgcrypto functions were non-strict and checked for NULLs. > >>> In 8.1 they were made STRICT. > >>> In 8.2 the NULL check were removed from code. > > > >> Not an smart move IMHO, I didn't create the function, it was created > >> as part of my upgrade process. May I suggest to put back the check?. > > > > That's the standard way of doing things in C functions (ie, rely on > > STRICT markings) and I see nothing wrong with it. > > > > If you were using an upgrade process that failed to incorporate > > version-to-version changes in the definitions of contrib functions, > > this is hardly going to be the only problem you encounter. > > I was under the impression that the standar procedure for upgrading > was to read the release notes searching for this kind of gotchas. I've > reviewed (again) the release notes for 8.1 and 8.2 and didn't find > anything related but maybe I'm not searching hard enough. > > If something is not going a work (or is going a work in a different > way) in some version after loading a shot form a previous one I think > it should be documented or some kind of backwards compatibility > mechanism should be provided. That's a valid complaint and I take the blame. The problem was that such "evolution" was not actually planned. So when I noticed the 8.2 commit, I did not think of the implications hard enough to realize the need for release note for it. Tom, how about putting a note about that into next 8.2 minor release notes? (8.3 too?) Something like "You need to refresh pgcrypto functions, because since rel 8.2 the code depends on functions being tagged STRICT." -- marko
Marko Kreen wrote: > On 8/24/07, Manuel Sugawara <masm@fciencias.unam.mx> wrote: > > Tom Lane <tgl@sss.pgh.pa.us> writes: > > > Manuel Sugawara <masm@fciencias.unam.mx> writes: > > >> "Marko Kreen" <markokr@gmail.com> writes: > > >>> In 8.0 the pgcrypto functions were non-strict and checked for NULLs. > > >>> In 8.1 they were made STRICT. > > >>> In 8.2 the NULL check were removed from code. > > > > > >> Not an smart move IMHO, I didn't create the function, it was created > > >> as part of my upgrade process. May I suggest to put back the check?. > > > > > > That's the standard way of doing things in C functions (ie, rely on > > > STRICT markings) and I see nothing wrong with it. > > > > > > If you were using an upgrade process that failed to incorporate > > > version-to-version changes in the definitions of contrib functions, > > > this is hardly going to be the only problem you encounter. > > > > I was under the impression that the standar procedure for upgrading > > was to read the release notes searching for this kind of gotchas. I've > > reviewed (again) the release notes for 8.1 and 8.2 and didn't find > > anything related but maybe I'm not searching hard enough. > > > > If something is not going a work (or is going a work in a different > > way) in some version after loading a shot form a previous one I think > > it should be documented or some kind of backwards compatibility > > mechanism should be provided. > > That's a valid complaint and I take the blame. > > The problem was that such "evolution" was not actually planned. > So when I noticed the 8.2 commit, I did not think of the implications > hard enough to realize the need for release note for it. > > Tom, how about putting a note about that into next 8.2 minor > release notes? (8.3 too?) Something like "You need to refresh > pgcrypto functions, because since rel 8.2 the code depends > on functions being tagged STRICT." Seems 8.2.5 was released without this release notes mention, but we haven't gotten any complaints about it so perhaps we don't need to add anything. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote: > Marko Kreen wrote: > > On 8/24/07, Manuel Sugawara <masm@fciencias.unam.mx> wrote: > > > If something is not going a work (or is going a work in a different > > > way) in some version after loading a shot form a previous one I think > > > it should be documented or some kind of backwards compatibility > > > mechanism should be provided. > > > > That's a valid complaint and I take the blame. > > > > The problem was that such "evolution" was not actually planned. > > So when I noticed the 8.2 commit, I did not think of the implications > > hard enough to realize the need for release note for it. > > > > Tom, how about putting a note about that into next 8.2 minor > > release notes? (8.3 too?) Something like "You need to refresh > > pgcrypto functions, because since rel 8.2 the code depends > > on functions being tagged STRICT." > > Seems 8.2.5 was released without this release notes mention, but we > haven't gotten any complaints about it so perhaps we don't need to add > anything. Huh, I see exactly that complaint above. -- Alvaro Herrera http://www.amazon.com/gp/registry/5ZYLFMCVHXC "Estoy de acuerdo contigo en que la verdad absoluta no existe... El problema es que la mentira sà existe y tu estás mintiendo" (G. Lama)
Alvaro Herrera wrote: > > > > If something is not going a work (or is going a work in a different > > > > way) in some version after loading a shot form a previous one I think > > > > it should be documented or some kind of backwards compatibility > > > > mechanism should be provided. > > > > > > That's a valid complaint and I take the blame. > > > > > > The problem was that such "evolution" was not actually planned. > > > So when I noticed the 8.2 commit, I did not think of the implications > > > hard enough to realize the need for release note for it. > > > > > > Tom, how about putting a note about that into next 8.2 minor > > > release notes? (8.3 too?) Something like "You need to refresh > > > pgcrypto functions, because since rel 8.2 the code depends > > > on functions being tagged STRICT." > > > > Seems 8.2.5 was released without this release notes mention, but we > > haven't gotten any complaints about it so perhaps we don't need to add > > anything. > > Huh, I see exactly that complaint above. My point is that we can't mention it in the release notes until 8.2.6. Will there still be people who are having an issue with it who haven't found the problem already? And if we put it in 8.2.6, it really was effective for 8.2 so we just mention it in 8.2.6 and say it applies to 8.2.X too? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On 11/6/07, Bruce Momjian <bruce@momjian.us> wrote: > Alvaro Herrera wrote: > > > > Tom, how about putting a note about that into next 8.2 minor > > > > release notes? (8.3 too?) Something like "You need to refresh > > > > pgcrypto functions, because since rel 8.2 the code depends > > > > on functions being tagged STRICT." > > > > > > Seems 8.2.5 was released without this release notes mention, but we > > > haven't gotten any complaints about it so perhaps we don't need to add > > > anything. > > > > Huh, I see exactly that complaint above. > > My point is that we can't mention it in the release notes until 8.2.6. > Will there still be people who are having an issue with it who haven't > found the problem already? And if we put it in 8.2.6, it really was > effective for 8.2 so we just mention it in 8.2.6 and say it applies to > 8.2.X too? We know that only few people have found the problem, because there is no way for them to guess what the fix should be by themselves, without consulting postgres lists. And I suspect most of the people who have not found out about the bug would still like to fix it in their setup, as crashing database is not fun. If only they knew about the problem... -- marko
Marko Kreen wrote: > On 11/6/07, Bruce Momjian <bruce@momjian.us> wrote: > > Alvaro Herrera wrote: > > > > > Tom, how about putting a note about that into next 8.2 minor > > > > > release notes? (8.3 too?) Something like "You need to refresh > > > > > pgcrypto functions, because since rel 8.2 the code depends > > > > > on functions being tagged STRICT." > > > > > > > > Seems 8.2.5 was released without this release notes mention, but we > > > > haven't gotten any complaints about it so perhaps we don't need to add > > > > anything. > > > > > > Huh, I see exactly that complaint above. > > > > My point is that we can't mention it in the release notes until 8.2.6. > > Will there still be people who are having an issue with it who haven't > > found the problem already? And if we put it in 8.2.6, it really was > > effective for 8.2 so we just mention it in 8.2.6 and say it applies to > > 8.2.X too? > > We know that only few people have found the problem, because > there is no way for them to guess what the fix should be > by themselves, without consulting postgres lists. > > And I suspect most of the people who have not found out about > the bug would still like to fix it in their setup, as crashing > database is not fun. If only they knew about the problem... OK, should we add a mention in the 8.2.6 release notes or put it in the 8. notes and figure if someone is upgrading they will read the 8.2 notes? What would the description be? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On 11/7/07, Bruce Momjian <bruce@momjian.us> wrote: > OK, should we add a mention in the 8.2.6 release notes or put it in the > 8. notes and figure if someone is upgrading they will read the 8.2 > notes? What would the description be? """ Note about pgcrypto - the functions were made STRICT in 8.1 and the NULL check was removed in 8.2 from C code. If some database still contains non-STRICT signatures from 8.0 and below, the functions will crash on NULL. To fix it the pgcrypto.sql needs to be run that will re-create the signatures. """ Hm. Is there way to say that better? -- marko
Marko Kreen wrote: > On 11/7/07, Bruce Momjian <bruce@momjian.us> wrote: > > OK, should we add a mention in the 8.2.6 release notes or put it in the > > 8. notes and figure if someone is upgrading they will read the 8.2 > > notes? What would the description be? > > """ > Note about pgcrypto - the functions were made STRICT in 8.1 > and the NULL check was removed in 8.2 from C code. If some > database still contains non-STRICT signatures from 8.0 and below, > the functions will crash on NULL. To fix it the pgcrypto.sql > needs to be run that will re-create the signatures. > """ > > Hm. Is there way to say that better? OK, first question. How would someone have the 8.0 signatures in 8.2? Don't they reinstall pgcrypto for every major release? -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
On 11/10/07, Bruce Momjian <bruce@momjian.us> wrote: > OK, first question. How would someone have the 8.0 signatures in 8.2? > Don't they reinstall pgcrypto for every major release? Why reinstall if the functions come from dump? At least in skype we stumbled on the problem. But if the reinstall is widespread practice, then indeed it may not be worth bothering. -- marko
"Marko Kreen" <markokr@gmail.com> writes:
> On 11/10/07, Bruce Momjian <bruce@momjian.us> wrote:
>> OK, first question. How would someone have the 8.0 signatures in 8.2?
>> Don't they reinstall pgcrypto for every major release?
> Why reinstall if the functions come from dump?
> At least in skype we stumbled on the problem. But if the reinstall
> is widespread practice, then indeed it may not be worth bothering.
If you don't re-run a contrib module's sql script then you don't get any
new functions that may have been added to the module. So I think the
real issue here is that we ought to deprecate the idea of skipping that
step, period.
regards, tom lane
Marko Kreen wrote: > On 11/10/07, Bruce Momjian <bruce@momjian.us> wrote: > > OK, first question. How would someone have the 8.0 signatures in 8.2? > > Don't they reinstall pgcrypto for every major release? > > Why reinstall if the functions come from dump? > > At least in skype we stumbled on the problem. But if the reinstall > is widespread practice, then indeed it may not be worth bothering. I assumed that the /contrib modules changed enough that people would always reinstall them for a major release. However, I don't know how to excluded them from dumps. If you don't have the pgcrypto library installed I think the CREATE FUNCTIONs fail because there is no library. The strange thing is we have never gotten too many complaints about upgrading the /contrib stuff, which is kind of odd. We have talked about putting each /contrib in its own schema so it could be easily excluded but that never happened. Putting all the contribs in one schema might be a good middle ground. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Tom Lane wrote: > "Marko Kreen" <markokr@gmail.com> writes: > > On 11/10/07, Bruce Momjian <bruce@momjian.us> wrote: > >> OK, first question. How would someone have the 8.0 signatures in 8.2? > >> Don't they reinstall pgcrypto for every major release? > > > Why reinstall if the functions come from dump? > > > At least in skype we stumbled on the problem. But if the reinstall > > is widespread practice, then indeed it may not be worth bothering. > > If you don't re-run a contrib module's sql script then you don't get any > new functions that may have been added to the module. So I think the > real issue here is that we ought to deprecate the idea of skipping that > step, period. When did we ever promote such a skipping idea? Seems we need to add some documentation for this. Once we get the /contrib docs moved to SGML I can add something. The good news is that the scripts have CREATE OR REPLACE function so it would clean up everything to match the library you just installed. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian <bruce@momjian.us> writes:
> Tom Lane wrote:
>> If you don't re-run a contrib module's sql script then you don't get any
>> new functions that may have been added to the module. So I think the
>> real issue here is that we ought to deprecate the idea of skipping that
>> step, period.
> The good news is that the scripts have CREATE OR REPLACE function so it
> would clean up everything to match the library you just installed.
Hmm ... now that you mention it: some of them do, some of them don't.
And some of them wrap the whole script in BEGIN/COMMIT, which means that
a conflict against existing definitions would prevent any new
definitions from being added.
Seems like maybe we need an explicit project policy that contrib scripts
should be able to overwrite an existing older installation properly;
maybe with some error reports, but not failing outright.
If so, someone will have to do the legwork of really making that happen
... any volunteers?
regards, tom lane
Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > Tom Lane wrote: > >> If you don't re-run a contrib module's sql script then you don't get any > >> new functions that may have been added to the module. So I think the > >> real issue here is that we ought to deprecate the idea of skipping that > >> step, period. > > > The good news is that the scripts have CREATE OR REPLACE function so it > > would clean up everything to match the library you just installed. > > Hmm ... now that you mention it: some of them do, some of them don't. > And some of them wrap the whole script in BEGIN/COMMIT, which means that > a conflict against existing definitions would prevent any new > definitions from being added. > > Seems like maybe we need an explicit project policy that contrib scripts > should be able to overwrite an existing older installation properly; > maybe with some error reports, but not failing outright. > > If so, someone will have to do the legwork of really making that happen > ... any volunteers? I have cleaned up the contrib install/uninstall to be more consistent, used CREATE OR REPLACE function consisently, and removed transaction blocks. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian wrote: > Tom Lane wrote: > > "Marko Kreen" <markokr@gmail.com> writes: > > > On 11/10/07, Bruce Momjian <bruce@momjian.us> wrote: > > >> OK, first question. How would someone have the 8.0 signatures in 8.2? > > >> Don't they reinstall pgcrypto for every major release? > > > > > Why reinstall if the functions come from dump? > > > > > At least in skype we stumbled on the problem. But if the reinstall > > > is widespread practice, then indeed it may not be worth bothering. > > > > If you don't re-run a contrib module's sql script then you don't get any > > new functions that may have been added to the module. So I think the > > real issue here is that we ought to deprecate the idea of skipping that > > step, period. > > When did we ever promote such a skipping idea? Seems we need to add > some documentation for this. Once we get the /contrib docs moved to > SGML I can add something. Documentation added. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +
Bruce Momjian <bruce@momjian.us> writes:
> I have cleaned up the contrib install/uninstall to be more consistent,
> used CREATE OR REPLACE function consisently, and removed transaction
> blocks.
... and broken the buildfarm ...
regards, tom lane
Tom Lane wrote: > Bruce Momjian <bruce@momjian.us> writes: > > I have cleaned up the contrib install/uninstall to be more consistent, > > used CREATE OR REPLACE function consisently, and removed transaction > > blocks. > > ... and broken the buildfarm ... OK, fixed, thanks. -- Bruce Momjian <bruce@momjian.us> http://momjian.us EnterpriseDB http://postgres.enterprisedb.com + If your life is a hard drive, Christ can be your backup. +