optimizer cost calculation problem - Mailing list pgsql-hackers

Kenji Sugita has identified a problem with cost_sort() in costsize.c.

In the following code fragment, sortmembytes is defined as long. So
    double        nruns = nbytes / (sortmembytes * 2);

may cause an integer overflow if sortmembytes exceeds 2^30, which in
turn make optimizer to produce wrong query plan(this actually happned
in a large PostgreSQL installation which has tons of memory). Here is a
proposed fix against current:

*** optimizer/path/costsize.c    16 Feb 2003 02:30:38 -0000    1.107
--- optimizer/path/costsize.c    31 Mar 2003 22:25:06 -0000
***************
*** 490,496 ****     Cost        startup_cost = input_cost;     Cost        run_cost = 0;     double        nbytes =
relation_byte_size(tuples,width);
 
!     long        sortmembytes = SortMem * 1024L;      if (!enable_sort)         startup_cost += disable_cost;
--- 490,496 ----     Cost        startup_cost = input_cost;     Cost        run_cost = 0;     double        nbytes =
relation_byte_size(tuples,width);
 
!     double        sortmembytes = SortMem * 1024.0;      if (!enable_sort)         startup_cost += disable_cost;