Re: Linux/PostgreSQL scalability issue - problem with 8 cores - Mailing list pgsql-performance

Hi Tom & all,

 >> It sounds a bit like momentary contention for a spinlock,
 >> but exactly what isn't clear.

 > ok, we're going to try oprofile, will let you know...

yes, it seems like contention for spinlock if I'm intepreting oprofile
correctly, around 60% of time during spikes is in s_lock. [for details
see below].

We've tried several times to get stacktrace from some of the running
backends during spikes, we got always this:

0x00002b005d00a9a9 in semop () from /lib/libc.so.6
#0  0x00002b005d00a9a9 in semop () from /lib/libc.so.6
#1  0x000000000054fe53 in PGSemaphoreLock (sema=0x2b00a04e5090,
interruptOK=0 '\0') at pg_sema.c:411
#2  0x0000000000575d95 in LWLockAcquire (lockid=SInvalLock,
mode=LW_EXCLUSIVE) at lwlock.c:455
#3  0x000000000056fbfe in ReceiveSharedInvalidMessages
(invalFunction=0x5e9a30 <LocalExecuteInvalidationMessage>,
    resetFunction=0x5e9df0 <InvalidateSystemCaches>) at sinval.c:159
#4  0x0000000000463505 in StartTransactionCommand () at xact.c:1439
#5  0x000000000056fa4b in ProcessCatchupEvent () at sinval.c:347
#6  0x000000000056fb20 in CatchupInterruptHandler
(postgres_signal_arg=<value optimized out>) at sinval.c:221
#7  0x00002b005cf6f110 in killpg () from /lib/libc.so.6
#8  0x0000000000000000 in ?? ()

Is this enough info to guess what's happening? What should we try next?

Thanks,

Kuba

oprofile results:

[I've shortened path from /usr/local/pg... to just pg for better
readilibity]

# opreport --long-filenames
CPU: Core 2, speed 2666.76 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
   125577 90.7584 pg-8.2.4/bin/postgres
     3792  2.7406 /lib/libc-2.3.6.so
     3220  2.3272 /usr/src/linux-2.6.22.15/vmlinux
     2145  1.5503 /usr/bin/oprofiled
     1540  1.1130 /xfs
      521  0.3765 pg-8.2.4/lib/plpgsql.so
      441  0.3187 /cciss
      374  0.2703 /oprofile

...

CPU: Core 2, speed 2666.76 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
unit mask of 0x00 (Unhalted core cycles) count 100000
samples  %        app name                 symbol name
85355    61.6887  pg-8.2.4/bin/postgres s_lock
9803      7.0849  pg-8.2.4/bin/postgres LWLockRelease
5535      4.0003  pg-8.2.4/bin/postgres LWLockAcquire
3792      2.7406  /lib/libc-2.3.6.so       (no symbols)
3724      2.6915  pg-8.2.4/bin/postgres DropRelFileNodeBuffers
2145      1.5503  /usr/bin/oprofiled       (no symbols)
2069      1.4953  pg-8.2.4/bin/postgres GetSnapshotData
1540      1.1130  /xfs                     (no symbols)
1246      0.9005  pg-8.2.4/bin/postgres hash_search_with_hash_value
1052      0.7603  pg-8.2.4/bin/postgres AllocSetAlloc
1015      0.7336  pg-8.2.4/bin/postgres heapgettup
879       0.6353  pg-8.2.4/bin/postgres hash_any
862       0.6230  /usr/src/linux-2.6.22.15/vmlinux mwait_idle
740       0.5348  pg-8.2.4/bin/postgres hash_seq_search
674       0.4871  pg-8.2.4/bin/postgres HeapTupleSatisfiesNow
557       0.4026  pg-8.2.4/bin/postgres SIGetDataEntry
552       0.3989  pg-8.2.4/bin/postgres equal
469       0.3390  pg-8.2.4/bin/postgres SearchCatCache
441       0.3187  /cciss                   (no symbols)
433       0.3129  /usr/src/linux-2.6.22.15/vmlinux find_busiest_group
413       0.2985  pg-8.2.4/bin/postgres PinBuffer
393       0.2840  pg-8.2.4/bin/postgres MemoryContextAllocZeroAligned
374       0.2703  /oprofile                (no symbols)
275       0.1988  pg-8.2.4/bin/postgres ExecInitExpr
253       0.1829  pg-8.2.4/bin/postgres base_yyparse
206       0.1489  pg-8.2.4/bin/postgres CatalogCacheFlushRelation
201       0.1453  pg-8.2.4/bin/postgres MemoryContextAlloc
194       0.1402  pg-8.2.4/bin/postgres _bt_compare
188       0.1359  /nf_conntrack            (no symbols)
158       0.1142  /bnx2                    (no symbols)
147       0.1062  pg-8.2.4/bin/postgres pgstat_initstats
139       0.1005  pg-8.2.4/bin/postgres fmgr_info_cxt_security
132       0.0954  /usr/src/linux-2.6.22.15/vmlinux task_rq_lock
131       0.0947  /bin/bash                (no symbols)
129       0.0932  pg-8.2.4/bin/postgres AllocSetFree
125       0.0903  pg-8.2.4/bin/postgres ReadBuffer
124       0.0896  pg-8.2.4/bin/postgres MemoryContextCreate
124       0.0896  pg-8.2.4/bin/postgres SyncOneBuffer
124       0.0896  pg-8.2.4/bin/postgres XLogInsert
123       0.0889  pg-8.2.4/bin/postgres _equalAggref
122       0.0882  pg-8.2.4/bin/postgres HeapTupleSatisfiesSnapshot
112       0.0809  pg-8.2.4/bin/postgres copyObject
102       0.0737  pg-8.2.4/bin/postgres UnpinBuffer
99        0.0716  pg-8.2.4/bin/postgres _SPI_execute_plan
99        0.0716  pg-8.2.4/bin/postgres nocachegetattr
98        0.0708  /usr/src/linux-2.6.22.15/vmlinux __wake_up_bit
97        0.0701  pg-8.2.4/bin/postgres TransactionIdIsInProgress
94        0.0679  pg-8.2.4/bin/postgres check_stack_depth
93        0.0672  pg-8.2.4/bin/postgres base_yylex
91        0.0658  pg-8.2.4/bin/postgres pfree
89        0.0643  pg-8.2.4/lib/plpgsql.so exec_stmts
86        0.0622  /usr/src/linux-2.6.22.15/vmlinux __switch_to
85        0.0614  pg-8.2.4/bin/postgres LockAcquire
83        0.0600  pg-8.2.4/bin/postgres FunctionCall2
82        0.0593  pg-8.2.4/bin/postgres ExecInitAgg
82        0.0593  /usr/src/linux-2.6.22.15/vmlinux system_call
79        0.0571  pg-8.2.4/bin/postgres pgstat_write_statsfile
77        0.0557  pg-8.2.4/bin/postgres heap_getsysattr
73        0.0528  pg-8.2.4/bin/postgres .plt
72        0.0520  /lib/ld-2.3.6.so         (no symbols)
71        0.0513  pg-8.2.4/bin/postgres SearchSysCache
71        0.0513  pg-8.2.4/bin/postgres _bt_checkkeys
68        0.0491  /usr/src/linux-2.6.22.15/vmlinux apic_timer_interrupt
67        0.0484  pg-8.2.4/bin/postgres slot_deform_tuple
64        0.0463  pg-8.2.4/bin/postgres TupleDescInitEntry
64        0.0463  pg-8.2.4/bin/postgres newarc
63        0.0455  /usr/src/linux-2.6.22.15/vmlinux __wake_up
62        0.0448  pg-8.2.4/bin/postgres LocalExecuteInvalidationMessage
61        0.0441  /usr/src/linux-2.6.22.15/vmlinux try_to_wake_up
60        0.0434  pg-8.2.4/bin/postgres ReceiveSharedInvalidMessages
59        0.0426  pg-8.2.4/bin/postgres ExecutorStart
58        0.0419  pg-8.2.4/bin/postgres DirectFunctionCall1
58        0.0419  pg-8.2.4/bin/postgres ScanKeywordLookup
57        0.0412  pg-8.2.4/bin/postgres hash_search
56        0.0405  pg-8.2.4/bin/postgres CatalogCacheComputeHashValue
56        0.0405  pg-8.2.4/bin/postgres ExecProject
55        0.0398  pg-8.2.4/bin/postgres _bt_first
54        0.0390  pg-8.2.4/lib/plpgsql.so exec_eval_simple_expr
53        0.0383  pg-8.2.4/bin/postgres AllocSetDelete
53        0.0383  pg-8.2.4/bin/postgres CleanupTempFiles
52        0.0376  pg-8.2.4/bin/postgres ExecCreateTupleTable
52        0.0376  pg-8.2.4/lib/plpgsql.so copy_plpgsql_datum
49        0.0354  pg-8.2.4/bin/postgres MemoryContextAllocZero
49        0.0354  pg-8.2.4/bin/postgres SIDelExpiredDataEntries
49        0.0354  pg-8.2.4/bin/postgres fix_opfuncids_walker
47        0.0340  pg-8.2.4/bin/postgres expression_tree_walker
46        0.0332  pg-8.2.4/bin/postgres LockBuffer
45        0.0325  pg-8.2.4/bin/postgres lappend
45        0.0325  /usr/src/linux-2.6.22.15/vmlinux do_IRQ
44        0.0318  pg-8.2.4/bin/postgres LockReleaseAll
43        0.0311  pg-8.2.4/bin/postgres ExecutorRun
43        0.0311  pg-8.2.4/bin/postgres exprTypmod
42        0.0304  pg-8.2.4/bin/postgres ExecClearTuple
42        0.0304  pg-8.2.4/bin/postgres heap_fill_tuple
41        0.0296  pg-8.2.4/bin/postgres ExecIndexBuildScanKeys
40        0.0289  pg-8.2.4/bin/postgres _bt_readpage
40        0.0289  pg-8.2.4/lib/plpgsql.so plpgsql_exec_function
40        0.0289  /usr/src/linux-2.6.22.15/vmlinux lock_timer_base
39        0.0282  pg-8.2.4/bin/postgres heap_release_fetch
38        0.0275  pg-8.2.4/bin/postgres ExecEvalVar
38        0.0275  pg-8.2.4/bin/postgres ExecTypeFromTLInternal
37        0.0267  pg-8.2.4/bin/postgres LockReassignCurrentOwner
37        0.0267  /usr/src/linux-2.6.22.15/vmlinux scheduler_tick
36        0.0260  pg-8.2.4/bin/postgres ReleaseAndReadBuffer
36        0.0260  pg-8.2.4/bin/postgres _bt_checkpage
36        0.0260  pg-8.2.4/bin/postgres freearc
36        0.0260  pg-8.2.4/bin/postgres heapgetpage
36        0.0260  /usr/src/linux-2.6.22.15/vmlinux resched_task
35        0.0253  pg-8.2.4/bin/postgres PGSemaphoreLock
35        0.0253  pg-8.2.4/bin/postgres optimize
35        0.0253  pg-8.2.4/bin/postgres slot_getattr
35        0.0253  /usr/src/linux-2.6.22.15/vmlinux effective_prio
35        0.0253  /usr/src/linux-2.6.22.15/vmlinux hrtimer_run_queues
34        0.0246  /ip_tables               (no symbols)
34        0.0246  pg-8.2.4/bin/postgres ExecCountSlotsNode
34        0.0246  pg-8.2.4/bin/postgres ExecMakeFunctionResultNoSets
34        0.0246  pg-8.2.4/bin/postgres ResourceOwnerForgetBuffer
34        0.0246  pg-8.2.4/bin/postgres TransactionIdPrecedes
34        0.0246  pg-8.2.4/bin/postgres _bt_getroot
34        0.0246  pg-8.2.4/lib/plpgsql.so exec_stmt_block
34        0.0246  /usr/src/linux-2.6.22.15/vmlinux do_page_fault
34        0.0246  /usr/src/linux-2.6.22.15/vmlinux run_rebalance_domains
33        0.0239  pg-8.2.4/bin/postgres AtEOXact_GUC
33        0.0239  /usr/src/linux-2.6.22.15/vmlinux __exit_idle
33        0.0239  /usr/src/linux-2.6.22.15/vmlinux sched_clock
32        0.0231  pg-8.2.4/bin/postgres _bt_moveright
32        0.0231  pg-8.2.4/bin/postgres compact
32        0.0231  /usr/src/linux-2.6.22.15/vmlinux __mod_timer
31        0.0224  pg-8.2.4/bin/postgres CreateExprContext
31        0.0224  pg-8.2.4/bin/postgres ExecInitNode
31        0.0224  pg-8.2.4/bin/postgres ExecMakeFunctionResult
31        0.0224  pg-8.2.4/bin/postgres ReleaseCatCache
31        0.0224  pg-8.2.4/bin/postgres exprType
30        0.0217  pg-8.2.4/bin/postgres _bt_binsrch
30        0.0217  pg-8.2.4/bin/postgres new_tail_cell
30        0.0217  /usr/src/linux-2.6.22.15/vmlinux run_timer_softirq
29        0.0210  pg-8.2.4/bin/postgres ExecAssignScanProjectionInfo
29        0.0210  pg-8.2.4/bin/postgres heap_form_tuple
28        0.0202  /usr/lib/gconv/ISO8859-1.so (no symbols)
28        0.0202  pg-8.2.4/bin/postgres AllocSetContextCreate
28        0.0202  pg-8.2.4/bin/postgres ResourceOwnerForgetCatCacheRef
28        0.0202  pg-8.2.4/bin/postgres hashoid
28        0.0202  pg-8.2.4/bin/postgres new_list
27        0.0195  pg-8.2.4/bin/postgres CreateTemplateTupleDesc
27        0.0195  pg-8.2.4/bin/postgres MemoryContextStrdup
27        0.0195  pg-8.2.4/bin/postgres SPI_execute_plan
27        0.0195  pg-8.2.4/bin/postgres btint4cmp
27        0.0195  /usr/src/linux-2.6.22.15/vmlinux IRQ0xa9_interrupt
26        0.0188  pg-8.2.4/bin/postgres ExecBuildProjectionInfo
26        0.0188  pg-8.2.4/bin/postgres ExecDropTupleTable
26        0.0188  pg-8.2.4/bin/postgres ExecEndNode
26        0.0188  pg-8.2.4/bin/postgres ExecEvalParam
26        0.0188  pg-8.2.4/bin/postgres FreeExecutorState
26        0.0188  pg-8.2.4/bin/postgres MemoryContextReset
26        0.0188  /usr/src/linux-2.6.22.15/vmlinux smp_apic_timer_interrupt
25        0.0181  pg-8.2.4/bin/postgres ExecInitSubPlan
25        0.0181  pg-8.2.4/bin/postgres FuncnameGetCandidates
25        0.0181  pg-8.2.4/bin/postgres MemoryContextDelete
25        0.0181  pg-8.2.4/bin/postgres SearchCatCacheList
25        0.0181  /usr/src/linux-2.6.22.15/vmlinux handle_edge_irq
25        0.0181  /usr/src/linux-2.6.22.15/vmlinux msecs_to_jiffies
24        0.0173  /nf_conntrack_ipv4       (no symbols)
24        0.0173  pg-8.2.4/bin/postgres ExecInitIndexScan
24        0.0173  pg-8.2.4/bin/postgres ExecProcNode
24        0.0173  pg-8.2.4/bin/postgres InternalCreateExecutorState
24        0.0173  pg-8.2.4/bin/postgres PostgresMain
24        0.0173  pg-8.2.4/bin/postgres RelationIncrementReferenceCount
24        0.0173  pg-8.2.4/bin/postgres heapgettup_pagemode
24        0.0173  /usr/src/linux-2.6.22.15/vmlinux __wake_up_common
24        0.0173  /usr/src/linux-2.6.22.15/vmlinux get_task_mm
24        0.0173  /usr/src/linux-2.6.22.15/vmlinux page_waitqueue
23        0.0166  pg-8.2.4/bin/postgres ExecCheckRTPerms
23        0.0166  pg-8.2.4/bin/postgres TransactionIdFollowsOrEquals
23        0.0166  pg-8.2.4/bin/postgres internal_putbytes
23        0.0166  pg-8.2.4/bin/postgres scanner_init
23        0.0166  pg-8.2.4/lib/plpgsql.so exec_stmt_execsql
23        0.0166  /usr/src/linux-2.6.22.15/vmlinux IRQ0xc1_interrupt
23        0.0166  /usr/src/linux-2.6.22.15/vmlinux __do_softirq
22        0.0159  pg-8.2.4/bin/postgres AfterTriggerBeginXact
22        0.0159  pg-8.2.4/bin/postgres GetTransactionSnapshot
22        0.0159  pg-8.2.4/bin/postgres ResourceOwnerReleaseInternal
22        0.0159  pg-8.2.4/bin/postgres lcons
22        0.0159  pg-8.2.4/bin/postgres markcanreach
22        0.0159  pg-8.2.4/lib/plpgsql.so exec_eval_datum
22        0.0159  pg-8.2.4/lib/plpgsql.so exec_eval_expr
22        0.0159  pg-8.2.4/lib/utf8_and_win.so win_to_utf8
22        0.0159  /usr/src/linux-2.6.22.15/vmlinux enqueue_task
22        0.0159  /usr/src/linux-2.6.22.15/vmlinux math_state_restore
21        0.0152  pg-8.2.4/bin/postgres AtCommit_Notify
21        0.0152  pg-8.2.4/bin/postgres CreateTupleDescCopy
21        0.0152  pg-8.2.4/bin/postgres ExecScan
21        0.0152  pg-8.2.4/bin/postgres ResourceOwnerEnlargeCatCacheRefs
21        0.0152  pg-8.2.4/bin/postgres ResourceOwnerForgetRelationRef
21        0.0152  pg-8.2.4/bin/postgres ResourceOwnerRememberCatCacheRef
21        0.0152  pg-8.2.4/bin/postgres SHMQueueInsertBefore
21        0.0152  pg-8.2.4/bin/postgres exec_simple_query
21        0.0152  pg-8.2.4/bin/postgres get_hash_value
21        0.0152  pg-8.2.4/bin/postgres index_getnext
21        0.0152  pg-8.2.4/bin/postgres next_token
21        0.0152  pg-8.2.4/bin/postgres pgstat_report_activity
21        0.0152  pg-8.2.4/lib/plpgsql.so exec_run_select
20        0.0145  pg-8.2.4/bin/postgres AllocSetReset
20        0.0145  pg-8.2.4/bin/postgres AtProcExit_Buffers
20        0.0145  pg-8.2.4/bin/postgres LocalToUtf
20        0.0145  pg-8.2.4/bin/postgres index_getprocinfo
20        0.0145  pg-8.2.4/bin/postgres spi_dest_startup
20        0.0145  pg-8.2.4/bin/postgres tag_hash
20        0.0145  /usr/src/linux-2.6.22.15/vmlinux sysret_check
19        0.0137  pg-8.2.4/bin/postgres Async_Unlisten
19        0.0137  pg-8.2.4/bin/postgres FileSeek
19        0.0137  pg-8.2.4/bin/postgres PortalStart
19        0.0137  pg-8.2.4/bin/postgres heap_open
19        0.0137  pg-8.2.4/bin/postgres pg_mblen
19        0.0137  /usr/src/linux-2.6.22.15/vmlinux prepare_to_wait
18        0.0130  /iptable_filter          (no symbols)
18        0.0130  /iptable_nat             (no symbols)
18        0.0130  /lib/libm-2.3.6.so       (no symbols)
18        0.0130  pg-8.2.4/bin/postgres CopySnapshot
18        0.0130  pg-8.2.4/bin/postgres ExecSetSlotDescriptor
18        0.0130  pg-8.2.4/bin/postgres SPI_connect
18        0.0130  pg-8.2.4/bin/postgres cleartraverse
18        0.0130  pg-8.2.4/bin/postgres pg_mbcliplen
18        0.0130  pg-8.2.4/bin/postgres strlcpy
18        0.0130  pg-8.2.4/lib/plpgsql.so plpgsql_compile
18        0.0130  /usr/src/linux-2.6.22.15/vmlinux account_system_time
18        0.0130  /usr/src/linux-2.6.22.15/vmlinux bit_waitqueue
17        0.0123  pg-8.2.4/bin/postgres AtEOXact_MultiXact
17        0.0123  pg-8.2.4/bin/postgres CreateTupleDescCopyConstr
17        0.0123  pg-8.2.4/bin/postgres ExecAgg
17        0.0123  pg-8.2.4/bin/postgres ExecResult
17        0.0123  pg-8.2.4/bin/postgres FlushBuffer
17        0.0123  pg-8.2.4/bin/postgres RelationGetIndexScan
17        0.0123  pg-8.2.4/bin/postgres SHMQueueDelete
17        0.0123  pg-8.2.4/bin/postgres SendRowDescriptionMessage
17        0.0123  pg-8.2.4/bin/postgres _bt_search
17        0.0123  pg-8.2.4/bin/postgres _copyTargetEntry
17        0.0123  pg-8.2.4/bin/postgres heap_getnext
17        0.0123  pg-8.2.4/bin/postgres markreachable
17        0.0123  pg-8.2.4/bin/postgres miss
17        0.0123  pg-8.2.4/bin/postgres scanRTEForColumn
17        0.0123  pg-8.2.4/lib/plpgsql.so exec_assign_value
17        0.0123  /usr/src/linux-2.6.22.15/vmlinux do_softirq
16        0.0116  pg-8.2.4/bin/postgres ExecEvalConst
16        0.0116  pg-8.2.4/bin/postgres ExecReScan
16        0.0116  pg-8.2.4/bin/postgres ExecSetParamPlan
16        0.0116  pg-8.2.4/bin/postgres downcase_truncate_identifier
16        0.0116  pg-8.2.4/bin/postgres transformExpr
16        0.0116  pg-8.2.4/bin/postgres varstr_cmp
16        0.0116  pg-8.2.4/lib/plpgsql.so plpgsql_call_handler
16        0.0116  pg-8.2.4/lib/plpgsql.so plpgsql_estate_setup
16        0.0116  /usr/src/linux-2.6.22.15/vmlinux cpu_idle
15        0.0108  pg-8.2.4/bin/postgres CommitTransactionCommand
15        0.0108  pg-8.2.4/bin/postgres ExecEvalFuncArgs
15        0.0108  pg-8.2.4/bin/postgres GrantLockLocal
15        0.0108  pg-8.2.4/bin/postgres PrepareToInvalidateCacheTuple
15        0.0108  pg-8.2.4/bin/postgres UtfToLocal
15        0.0108  pg-8.2.4/bin/postgres cost_nestloop
15        0.0108  pg-8.2.4/bin/postgres errstart
15        0.0108  pg-8.2.4/bin/postgres index_open
15        0.0108  pg-8.2.4/bin/postgres init_fcache
15        0.0108  pg-8.2.4/bin/postgres list_delete_cell
14        0.0101  pg-8.2.4/bin/postgres ExecStoreTuple
14        0.0101  pg-8.2.4/bin/postgres ReadNewTransactionId
14        0.0101  pg-8.2.4/bin/postgres ReleaseBuffer
14        0.0101  pg-8.2.4/bin/postgres ResourceOwnerRememberBuffer
14        0.0101  pg-8.2.4/bin/postgres SHMQueueNext
14        0.0101  pg-8.2.4/bin/postgres btgettuple
14        0.0101  pg-8.2.4/bin/postgres duptraverse
14        0.0101  pg-8.2.4/bin/postgres heap_compute_data_size
14        0.0101  pg-8.2.4/bin/postgres index_rescan
14        0.0101  pg-8.2.4/bin/postgres list_copy
14        0.0101  pg-8.2.4/bin/postgres smgrclosenode
14        0.0101  pg-8.2.4/lib/plpgsql.so .plt
14        0.0101  /usr/src/linux-2.6.22.15/vmlinux idle_cpu
14        0.0101  /usr/src/linux-2.6.22.15/vmlinux run_workqueue
13        0.0094  pg-8.2.4/bin/postgres Async_Listen
13        0.0094  pg-8.2.4/bin/postgres DLMoveToFront
13        0.0094  pg-8.2.4/bin/postgres FreeExprContext
13        0.0094  pg-8.2.4/bin/postgres IndexNext
13        0.0094  pg-8.2.4/bin/postgres LockRelease
13        0.0094  pg-8.2.4/bin/postgres ResourceOwnerEnlargeBuffers
13        0.0094  pg-8.2.4/bin/postgres ShutdownExprContext
13        0.0094  pg-8.2.4/bin/postgres cleanup
13        0.0094  pg-8.2.4/bin/postgres datumCopy
13        0.0094  pg-8.2.4/bin/postgres grouping_planner
13        0.0094  pg-8.2.4/bin/postgres newstate
13        0.0094  pg-8.2.4/bin/postgres pgstat_start
13        0.0094  pg-8.2.4/bin/postgres spi_printtup
13        0.0094  pg-8.2.4/bin/postgres transformStmt
12        0.0087  pg-8.2.4/bin/postgres ExecEvalOper
12        0.0087  pg-8.2.4/bin/postgres ExecQual
12        0.0087  pg-8.2.4/bin/postgres IsSharedRelation
12        0.0087  pg-8.2.4/bin/postgres RelationIdGetRelation
12        0.0087  pg-8.2.4/bin/postgres StartTransactionCommand
12        0.0087  pg-8.2.4/bin/postgres _bt_insertonpg
12        0.0087  pg-8.2.4/bin/postgres advance_aggregates
12        0.0087  pg-8.2.4/bin/postgres getvacant
12        0.0087  pg-8.2.4/bin/postgres list_delete_ptr
12        0.0087  pg-8.2.4/bin/postgres namestrcpy
12        0.0087  pg-8.2.4/bin/postgres oideq
12        0.0087  /usr/src/linux-2.6.22.15/vmlinux local_bh_enable_ip
12        0.0087  /usr/src/linux-2.6.22.15/vmlinux sys_rt_sigreturn
11        0.0080  pg-8.2.4/bin/postgres CommitTransaction
11        0.0080  pg-8.2.4/bin/postgres ExecEvalNullTest
11        0.0080  pg-8.2.4/bin/postgres LockRelationOid
11        0.0080  pg-8.2.4/bin/postgres ProcessUtility
11        0.0080  pg-8.2.4/bin/postgres UnGrantLock
11        0.0080  pg-8.2.4/bin/postgres XidInSnapshot
11        0.0080  pg-8.2.4/bin/postgres _bt_doinsert
11        0.0080  pg-8.2.4/bin/postgres _bt_steppage
11        0.0080  pg-8.2.4/bin/postgres appendBinaryStringInfo
11        0.0080  pg-8.2.4/bin/postgres okcolors
11        0.0080  pg-8.2.4/bin/postgres pq_getmessage
11        0.0080  pg-8.2.4/lib/plpgsql.so exec_move_row
11        0.0080  /usr/src/linux-2.6.22.15/vmlinux clocksource_get_next
11        0.0080  /usr/src/linux-2.6.22.15/vmlinux handle_IRQ_event
11        0.0080  /usr/src/linux-2.6.22.15/vmlinux local_bh_enable
11        0.0080  /usr/src/linux-2.6.22.15/vmlinux mod_timer
10        0.0072  pg-8.2.4/bin/postgres CommandEndInvalidationMessages
10        0.0072  pg-8.2.4/bin/postgres CopyTriggerDesc
10        0.0072  pg-8.2.4/bin/postgres ExecAssignExprContext
10        0.0072  pg-8.2.4/bin/postgres ExecEvalScalarArrayOp
10        0.0072  pg-8.2.4/bin/postgres FunctionCall5
10        0.0072  pg-8.2.4/bin/postgres GetCurrentSubTransactionId
10        0.0072  pg-8.2.4/bin/postgres RecordTransactionCommit
10        0.0072  pg-8.2.4/bin/postgres StrategyGetBuffer
10        0.0072  pg-8.2.4/bin/postgres _bt_getbuf
10        0.0072  pg-8.2.4/bin/postgres _bt_preprocess_keys
10        0.0072  pg-8.2.4/bin/postgres eval_const_expressions_mutator
10        0.0072  pg-8.2.4/bin/postgres index_beginscan_internal
10        0.0072  pg-8.2.4/bin/postgres new_head_cell
10        0.0072  pg-8.2.4/bin/postgres perform_default_encoding_conversion
10        0.0072  pg-8.2.4/bin/postgres pg_regcomp
10        0.0072  pg-8.2.4/lib/plpgsql.so exec_eval_boolean
10        0.0072  pg-8.2.4/lib/plpgsql.so exec_stmt_if
10        0.0072  /usr/src/linux-2.6.22.15/vmlinux __rcu_pending
10        0.0072  /usr/src/linux-2.6.22.15/vmlinux __rcu_process_callbacks