Thread: Server process crash - Segmentation fault
Hello. I was running PostgreSQL 9.1.4 when I got a server process crash (Segmentation fault) as the postgres log shown below.I tried upgrade to newest version 9.3.4, but this gives exactly the same problem. It is an (ecpg based) C-program that does tons of these scroll cursor exercises. Until recently this worked too but changesto totally different part of the program made this happen. (I have made way too many changes to this other part tobe able to "roll back" the code :-( ). The system generates data all the time for this lookup, but I can grab the SQL fromthe postgres log and run it through psql and get the result I expect, so I don't see how it can be data related. Please help, Leif . . . 22864 2014-05-06 15:37:35.350 CEST LOG: statement: close execcurs 22864 2014-05-06 15:37:35.350 CEST LOG: statement: deallocate "ApplDBConn_22854_f6adeb70_query" 22864 2014-05-06 15:37:35.352 CEST DEBUG: parse ApplDBConn_22854_f6adeb70_query: SELECT data_type FROM information_schema.columnsWHERE table_name = 'l2_hb_water_hours_sum' AND column_name = ''; 22864 2014-05-06 15:37:35.353 CEST LOG: statement: declare execcurs scroll cursor for SELECT data_type FROM information_schema.columnsWHERE table_name = 'l2_hb_water_hours_sum' AND column_name = ' '; 22864 2014-05-06 15:37:35.356 CEST LOG: statement: fetch first in execcurs 22864 2014-05-06 15:37:35.358 CEST LOG: statement: close execcurs 22864 2014-05-06 15:37:35.358 CEST LOG: statement: deallocate "ApplDBConn_22854_f6adeb70_query" 22864 2014-05-06 15:37:35.359 CEST LOG: statement: commit 22864 2014-05-06 15:37:35.359 CEST LOG: statement: start transaction read only 22864 2014-05-06 15:37:35.360 CEST DEBUG: parse ApplDBConn_22854_f6adeb70_query: SELECT montime, year, month, day, hh, gal_hour,exp_hour, unsched_hour FROM l2_hb_water_hours_sum WHERE l2_hb_water_ hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET (SELECT CASE WHEN count(*) > 2000 THEN count(*) -2000 ELSE 0 ENDFROM l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 86 ); 22864 2014-05-06 15:37:35.365 CEST LOG: statement: declare execcurs scroll cursor for SELECT montime, year, month, day,hh, gal_hour, exp_hour, unsched_hour FROM l2_hb_water_hours_sum WHERE l2_hb _water_hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET (SELECT CASE WHEN count(*) > 2000 THEN count(*) -2000 ELSE0 END FROM l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 8 6 ); 22864 2014-05-06 15:37:35.432 CEST LOG: statement: fetch first in execcurs 21702 2014-05-06 15:37:35.440 CEST DEBUG: server process (PID 22864) was terminated by signal 11: Segmentation fault 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was running: fetch first in execcurs 21702 2014-05-06 15:37:35.440 CEST LOG: server process (PID 22864) was terminated by signal 11: Segmentation fault 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was running: fetch first in execcurs 21702 2014-05-06 15:37:35.440 CEST LOG: terminating any other active server processes 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22896 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22893 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22882 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22751 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22749 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 22748 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 21705 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 21704 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 21706 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 21707 22751 2014-05-06 15:37:35.440 CEST WARNING: terminating connection because of crash of another server process 22751 2014-05-06 15:37:35.440 CEST DETAIL: The postmaster has commanded this server process to roll back the current transactionand exit, because another server process exited abnormally and possibl y corrupted shared memory. 22751 2014-05-06 15:37:35.440 CEST HINT: In a moment you should be able to reconnect to the database and repeat your command. 21702 2014-05-06 15:37:35.440 CEST DEBUG: sending SIGQUIT to process 21708
On 05/06/2014 07:08 AM, Leif Jensen wrote: > Hello. > > I was running PostgreSQL 9.1.4 when I got a server process crash (Segmentation fault) as the postgres log shown below.I tried upgrade to newest version 9.3.4, but this gives exactly the same problem. > > It is an (ecpg based) C-program that does tons of these scroll cursor exercises. Until recently this worked too butchanges to totally different part of the program made this happen. (I have made way too many changes to this other partto be able to "roll back" the code :-( ). The system generates data all the time for this lookup, but I can grab theSQL from the postgres log and run it through psql and get the result I expect, so I don't see how it can be data related. > > Please help, > > Leif > > . > . > . > 22864 2014-05-06 15:37:35.350 CEST LOG: statement: close execcurs > 22864 2014-05-06 15:37:35.350 CEST LOG: statement: deallocate "ApplDBConn_22854_f6adeb70_query" > 22864 2014-05-06 15:37:35.352 CEST DEBUG: parse ApplDBConn_22854_f6adeb70_query: SELECT data_type FROM information_schema.columnsWHERE table_name = 'l2_hb_water_hours_sum' AND column_name = ''; > 22864 2014-05-06 15:37:35.353 CEST LOG: statement: declare execcurs scroll cursor for SELECT data_type FROM information_schema.columnsWHERE table_name = 'l2_hb_water_hours_sum' AND column_name = ' > '; > 22864 2014-05-06 15:37:35.356 CEST LOG: statement: fetch first in execcurs > 22864 2014-05-06 15:37:35.358 CEST LOG: statement: close execcurs > 22864 2014-05-06 15:37:35.358 CEST LOG: statement: deallocate "ApplDBConn_22854_f6adeb70_query" > 22864 2014-05-06 15:37:35.359 CEST LOG: statement: commit > 22864 2014-05-06 15:37:35.359 CEST LOG: statement: start transaction read only > 22864 2014-05-06 15:37:35.360 CEST DEBUG: parse ApplDBConn_22854_f6adeb70_query: SELECT montime, year, month, day, hh,gal_hour, exp_hour, unsched_hour FROM l2_hb_water_hours_sum WHERE l2_hb_water_ > hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET (SELECT CASE WHEN count(*) > 2000 THEN count(*) -2000 ELSE 0END FROM l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 86 ); > 22864 2014-05-06 15:37:35.365 CEST LOG: statement: declare execcurs scroll cursor for SELECT montime, year, month, day,hh, gal_hour, exp_hour, unsched_hour FROM l2_hb_water_hours_sum WHERE l2_hb > _water_hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET (SELECT CASE WHEN count(*) > 2000 THEN count(*) -2000ELSE 0 END FROM l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 8 > 6 ); The code that generates the above would be helpful. The thing that catches my eye is that the first time you use ApplDBConn_22854_f6adeb70_query the parse and cursor queries are the same and all is good. The second time they are not and you get a failure. Without seeing what is going in in your code it is hard to tell if this significant or not. > 22864 2014-05-06 15:37:35.432 CEST LOG: statement: fetch first in execcurs > 21702 2014-05-06 15:37:35.440 CEST DEBUG: server process (PID 22864) was terminated by signal 11: Segmentation fault > 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was running: fetch first in execcurs > 21702 2014-05-06 15:37:35.440 CEST LOG: server process (PID 22864) was terminated by signal 11: Segmentation fault > 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was running: fetch first in execcurs > 21702 2014-05-06 15:37:35.440 CEST LOG: terminating any other active server processes -- Adrian Klaver adrian.klaver@aklaver.com
Hello Adrian, Thank you for your answer. I can post part of the code that makes these calls, but I'm not sure how much it would help.It is rather large function that makes these calls, and it is called all over the program. The part of the log postedis only a small excerpt of the use of the ApplDBConn_22854_f6adeb70_query, which has been used many many times beforethe log shown (167 in all to be exact ;-) ). Leif ----- Original Message ----- > On 05/06/2014 07:08 AM, Leif Jensen wrote: > > Hello. > > > > I was running PostgreSQL 9.1.4 when I got a server process crash > > (Segmentation fault) as the postgres log shown below. I tried > > upgrade to newest version 9.3.4, but this gives exactly the same > > problem. > > > > It is an (ecpg based) C-program that does tons of these scroll > > cursor exercises. Until recently this worked too but changes to > > totally different part of the program made this happen. (I have > > made way too many changes to this other part to be able to "roll > > back" the code :-( ). The system generates data all the time for > > this lookup, but I can grab the SQL from the postgres log and > > run it through psql and get the result I expect, so I don't see > > how it can be data related. > > > > Please help, > > > > Leif > > > > . > > . > > . > > 22864 2014-05-06 15:37:35.350 CEST LOG: statement: close execcurs > > 22864 2014-05-06 15:37:35.350 CEST LOG: statement: deallocate > > "ApplDBConn_22854_f6adeb70_query" > > 22864 2014-05-06 15:37:35.352 CEST DEBUG: parse > > ApplDBConn_22854_f6adeb70_query: SELECT data_type FROM > > information_schema.columns WHERE table_name = > > 'l2_hb_water_hours_sum' AND column_name = ''; > > 22864 2014-05-06 15:37:35.353 CEST LOG: statement: declare execcurs > > scroll cursor for SELECT data_type FROM information_schema.columns > > WHERE table_name = 'l2_hb_water_hours_sum' AND column_name = ' > > '; > > 22864 2014-05-06 15:37:35.356 CEST LOG: statement: fetch first in > > execcurs > > 22864 2014-05-06 15:37:35.358 CEST LOG: statement: close execcurs > > 22864 2014-05-06 15:37:35.358 CEST LOG: statement: deallocate > > "ApplDBConn_22854_f6adeb70_query" > > 22864 2014-05-06 15:37:35.359 CEST LOG: statement: commit > > 22864 2014-05-06 15:37:35.359 CEST LOG: statement: start transaction > > read only > > 22864 2014-05-06 15:37:35.360 CEST DEBUG: parse > > ApplDBConn_22854_f6adeb70_query: SELECT montime, year, month, day, > > hh, gal_hour, exp_hour, unsched_hour FROM l2_hb_water_hours_sum > > WHERE l2_hb_water_ > > hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET (SELECT CASE > > WHEN count(*) > 2000 THEN count(*) -2000 ELSE 0 END FROM > > l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 86 ); > > 22864 2014-05-06 15:37:35.365 CEST LOG: statement: declare execcurs > > scroll cursor for SELECT montime, year, month, day, hh, gal_hour, > > exp_hour, unsched_hour FROM l2_hb_water_hours_sum WHERE l2_hb > > _water_hours_sum.ctrlid = 86 ORDER BY year,month,day,hh OFFSET > > (SELECT CASE WHEN count(*) > 2000 THEN count(*) -2000 ELSE 0 END > > FROM l2_hb_water_hours_sum WHERE l2_hb_water_hours_sum.ctrlid = 8 > > 6 ); > > The code that generates the above would be helpful. The thing that > catches my eye is that the first time you use > ApplDBConn_22854_f6adeb70_query the parse and cursor queries are the > same and all is good. The second time they are not and you get a > failure. Without seeing what is going in in your code it is hard to > tell > if this significant or not. > > > 22864 2014-05-06 15:37:35.432 CEST LOG: statement: fetch first in > > execcurs > > 21702 2014-05-06 15:37:35.440 CEST DEBUG: server process (PID 22864) > > was terminated by signal 11: Segmentation fault > > 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was > > running: fetch first in execcurs > > 21702 2014-05-06 15:37:35.440 CEST LOG: server process (PID 22864) > > was terminated by signal 11: Segmentation fault > > 21702 2014-05-06 15:37:35.440 CEST DETAIL: Failed process was > > running: fetch first in execcurs > > 21702 2014-05-06 15:37:35.440 CEST LOG: terminating any other active > > server processes > > > -- > Adrian Klaver > adrian.klaver@aklaver.com
On 05/06/2014 11:37 PM, Leif Jensen wrote: > Hello Adrian, > > Thank you for your answer. I can post part of the code that makes these calls, but I'm not sure how much it would help.It is rather large function that makes these calls, and it is called all over the program. The part of the log postedis only a small excerpt of the use of the ApplDBConn_22854_f6adeb70_query, which has been used many many times beforethe log shown (167 in all to be exact ;-) ). Exactly. Something different happened at that point. The hard part will determining what that is. The next step would seem to run a debugger on the Postgres process to get more information. For a step by step guide see here: https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD > > Leif > > -- Adrian Klaver adrian.klaver@aklaver.com
Leif Jensen <leif@crysberg.dk> writes: > Thank you for your answer. I can post part of the code that makes these calls, but I'm not sure how much it would help.It is rather large function that makes these calls, and it is called all over the program. The part of the log postedis only a small excerpt of the use of the ApplDBConn_22854_f6adeb70_query, which has been used many many times beforethe log shown (167 in all to be exact ;-) ). Perhaps you could get a stack trace from the segfault, then. https://wiki.postgresql.org/wiki/Generating_a_stack_trace_of_a_PostgreSQL_backend regards, tom lane
Hi Guys, Here is a gdb dump of the backtrace at the server process crash. I have also included the code that generates these calls.As mentioned below this specific connection has been used many times before the crash. Also, we are aware of the threadcaveat that only using a connection from one thread at a time. Therefore the "strange" connection name that includesboth the process id and the thread id. This is for the code to make sure that a connection is only used in the threadit is meant to. Hope this can help in helping me ;-), Leif ----- Original Message ----- > On 05/06/2014 11:37 PM, Leif Jensen wrote: > > Hello Adrian, > > > > Thank you for your answer. I can post part of the code that > > makes these calls, but I'm not sure how much it would help. It > > is rather large function that makes these calls, and it is > > called all over the program. The part of the log posted is only > > a small excerpt of the use of the > > ApplDBConn_22854_f6adeb70_query, which has been used many many > > times before the log shown (167 in all to be exact ;-) ). > > Exactly. Something different happened at that point. The hard part > will > determining what that is. The next step would seem to run a debugger > on > the Postgres process to get more information. For a step by step guide > see here: > > https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD > > > > > Leif > > > > > > > > -- > Adrian Klaver > adrian.klaver@aklaver.com
Attachment
Leif Jensen <leif@crysberg.dk> writes: > Here is a gdb dump of the backtrace at the server process crash. I have also included the code that generates thesecalls. As mentioned below this specific connection has been used many times before the crash. Also, we are aware ofthe thread caveat that only using a connection from one thread at a time. Therefore the "strange" connection name thatincludes both the process id and the thread id. This is for the code to make sure that a connection is only used in thethread it is meant to. Hm. The crash looks like it must be because ActiveSnapshot is null (not set). Since we're doing a FETCH, the active snapshot ought to be the one saved for the cursor query by DECLARE CURSOR. It looks like the problem is that pquery.c only bothers to install that as the active snapshot while calling ExecutorRun, but in this stack trace we're in ExecutorRewind. I wonder if it's a bad idea for ExecReScanLimit to be executing user-defined expressions? But it's been like that for awhile, and I think we might have a hard time preserving the bounded-sort optimization if we didn't do that. Anyway the simple fix would be to ensure we install the query snapshot as active before calling ExecutorRewind. One interesting question is why this issue hasn't been seen before; it seems like it'd not be that hard to hit. regards, tom lane
Could it be related to the OFFSET part of the statement ? I have another query on the same table without OFFSET, whichseems to work fine. Leif ----- Original Message ----- > Leif Jensen <leif@crysberg.dk> writes: > > Here is a gdb dump of the backtrace at the server process crash. > > I have also included the code that generates these calls. As > > mentioned below this specific connection has been used many times > > before the crash. Also, we are aware of the thread caveat that > > only using a connection from one thread at a time. Therefore the > > "strange" connection name that includes both the process id and > > the thread id. This is for the code to make sure that a > > connection is only used in the thread it is meant to. > > Hm. The crash looks like it must be because ActiveSnapshot is null > (not set). Since we're doing a FETCH, the active snapshot ought to > be the one saved for the cursor query by DECLARE CURSOR. It looks > like the problem is that pquery.c only bothers to install that as the > active snapshot while calling ExecutorRun, but in this stack trace > we're in ExecutorRewind. > > I wonder if it's a bad idea for ExecReScanLimit to be executing > user-defined expressions? But it's been like that for awhile, > and I think we might have a hard time preserving the bounded-sort > optimization if we didn't do that. > > Anyway the simple fix would be to ensure we install the query > snapshot as active before calling ExecutorRewind. > > One interesting question is why this issue hasn't been seen before; > it seems like it'd not be that hard to hit. > > regards, tom lane
Leif Jensen <leif@crysberg.dk> writes: > Could it be related to the OFFSET part of the statement ? I have another query on the same table without OFFSET, whichseems to work fine. Yeah, the specific code path here involves executing a stable (or possibly immutable) SQL function in a LIMIT or OFFSET clause. I was able to reproduce the crash like so: create function foo(int) returns int as 'select $1 limit 1' language sql stable; begin; declare c cursor for select * from int8_tbl limit foo(3); select * from c; move backward all in c; select * from c; commit; You might be able to dodge the problem if you can make the SQL function inline-able (the LIMIT 1 in my example is just to prevent that from happening). A less appealing alternative is to mark the function VOLATILE, which I think would also prevent this crash, but might have negative performance consequences. If you don't mind building your own PG then you could grab the actual fix from our git repo; I should have something committed before long. regards, tom lane
Hi Tom, I already compiled postgreSQL myself and now using 9.3.4, so I would very much like a patch. Where can I find that ? Leif ----- Original Message ----- > Leif Jensen <leif@crysberg.dk> writes: > > Could it be related to the OFFSET part of the statement ? I have > > another query on the same table without OFFSET, which seems to > > work fine. > > Yeah, the specific code path here involves executing a stable (or > possibly > immutable) SQL function in a LIMIT or OFFSET clause. I was able to > reproduce the crash like so: > > create function foo(int) returns int as 'select $1 limit 1' > language sql stable; > > begin; > > declare c cursor for select * from int8_tbl limit foo(3); > > select * from c; > > move backward all in c; > > select * from c; > > commit; > > You might be able to dodge the problem if you can make the SQL > function > inline-able (the LIMIT 1 in my example is just to prevent that from > happening). A less appealing alternative is to mark the function > VOLATILE, which I think would also prevent this crash, but might have > negative performance consequences. > > If you don't mind building your own PG then you could grab the actual > fix > from our git repo; I should have something committed before long. > > regards, tom lane
Leif Jensen <leif@crysberg.dk> writes: > I already compiled postgreSQL myself and now using 9.3.4, so I would very much like a patch. Where can I find that ? 9.3 patch is here: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fc58c39d468587467c7c55b349c28167794eadaf regards, tom lane
On 05/08/2014 06:59 AM, Leif Jensen wrote: > Hi Tom, > > I already compiled postgreSQL myself and now using 9.3.4, so I would very much like a patch. Where can I find that? http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=04e5025be8bbe572e12b19c4ba9e2a8360b8ffe5 > > Leif -- Adrian Klaver adrian.klaver@aklaver.com
On 05/08/2014 07:19 AM, Tom Lane wrote: > Leif Jensen <leif@crysberg.dk> writes: >> I already compiled postgreSQL myself and now using 9.3.4, so I would very much like a patch. Where can I find that? > > 9.3 patch is here: > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fc58c39d468587467c7c55b349c28167794eadaf Alright, so I obviously linked to the wrong patch because I pointed at HEAD and not REL9_3_STABLE. What I am trying to figure out is what is the distinction between commit and commitdiff? > > regards, tom lane > > -- Adrian Klaver adrian.klaver@aklaver.com
Adrian Klaver <adrian.klaver@aklaver.com> writes: > On 05/08/2014 07:19 AM, Tom Lane wrote: >> 9.3 patch is here: >> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fc58c39d468587467c7c55b349c28167794eadaf > Alright, so I obviously linked to the wrong patch because I pointed at > HEAD and not REL9_3_STABLE. I think those patches are the same, actually, but I was trying to be careful. > What I am trying to figure out is what is > the distinction between commit and commitdiff? The "commitdiff" link shows you the actual diffs in the patch, the other one doesn't. In practice, Leif's going to want to hit the "patch" link anyway to get a clean downloadable patch; so likely what we should have pointed him at is http://git.postgresql.org/gitweb/?p=postgresql.git;a=patch;h=fc58c39d468587467c7c55b349c28167794eadaf I'm just in the habit of looking at the "commitdiff" versions of the web pages as being the best readability/information tradeoff for casual examination of a patch. regards, tom lane
On 05/08/2014 07:43 AM, Tom Lane wrote: > Adrian Klaver <adrian.klaver@aklaver.com> writes: >> On 05/08/2014 07:19 AM, Tom Lane wrote: >>> 9.3 patch is here: >>> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fc58c39d468587467c7c55b349c28167794eadaf > >> Alright, so I obviously linked to the wrong patch because I pointed at >> HEAD and not REL9_3_STABLE. > > I think those patches are the same, actually, but I was trying to be > careful. > >> What I am trying to figure out is what is >> the distinction between commit and commitdiff? > > The "commitdiff" link shows you the actual diffs in the patch, the other > one doesn't. Got it, saves opening the diff for each file. > > In practice, Leif's going to want to hit the "patch" link anyway to get a > clean downloadable patch; so likely what we should have pointed him at is > http://git.postgresql.org/gitweb/?p=postgresql.git;a=patch;h=fc58c39d468587467c7c55b349c28167794eadaf > I'm just in the habit of looking at the "commitdiff" versions of the web > pages as being the best readability/information tradeoff for casual > examination of a patch. Thanks for the explanation. > > regards, tom lane > > -- Adrian Klaver adrian.klaver@aklaver.com
Hello Tom, Adrian Thank you for your help and the patch. Things works nicely for me now :-). Leif ----- Original Message ----- > Adrian Klaver <adrian.klaver@aklaver.com> writes: > > On 05/08/2014 07:19 AM, Tom Lane wrote: > >> 9.3 patch is here: > >> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=fc58c39d468587467c7c55b349c28167794eadaf > > > Alright, so I obviously linked to the wrong patch because I pointed > > at > > HEAD and not REL9_3_STABLE. > > I think those patches are the same, actually, but I was trying to be > careful. > > > What I am trying to figure out is what is > > the distinction between commit and commitdiff? > > The "commitdiff" link shows you the actual diffs in the patch, the > other > one doesn't. > > In practice, Leif's going to want to hit the "patch" link anyway to > get a > clean downloadable patch; so likely what we should have pointed him at > is > http://git.postgresql.org/gitweb/?p=postgresql.git;a=patch;h=fc58c39d468587467c7c55b349c28167794eadaf > I'm just in the habit of looking at the "commitdiff" versions of the > web > pages as being the best readability/information tradeoff for casual > examination of a patch. > > regards, tom lane