We are currently working on a DBMS course project that requires us to modify postgresql optimizer. We need to vary work_mem within a range and note down the cost at those points(allocated memory) after 'explain' operation and then select the optimal tradeoff. We were changing "postgresql-9.1.12/src/backend/optimizer/plan/planner.c" where we created a new function mem_tradeoff() that calls planner() within a for loop for the varying memory. This mem_tradeoff() is called from pg_plan_query() in "postgresql-9.1.12/src/backend/tcop/postgres.c". The problem is, if the for loop runs only one iteration, everything is fine, but if we print plan->planTree->total_cost, it is printed 132 times for any query. Also if the for loop runs more than one iteration, we get a segmentation fault. Any help is very much appreciated.
