I just did a little experiment: extracted top four queries that were executed the longest during stall times and launched pgbench test with 240 clients. Yet I wasn't able to put the server into a stall with that. Also load average was hitting 120+, it was all user cpu, single digit % system. The system remained pretty responsive (on a keypress), in contrary to when high-sys-cpu stall happens, showing similar LA numbers.
This makes me think that there is probably a different (than ones I tried) query or condition that is responsible for creating high-sys-cpu. Also, from my limited knowledge of postgresql and kernel internals, this experiment puts under question idea that linux scheduler might be the corner stone....
Here is a longer log, first column is PID. High-sys-cpu stall happened around 10:42:37
I'll try to put pgbouncer in place and repeat, also though this will take longer time to setup.
