For example, please see the max latencies of test set 2 (PG 9.3) and test set 4 (xlog scaling with padding). They are 207.359 and 1219.422 respectively. The throughput is of course greatly improved, but I think the response time should not be sacrificed as much as possible. There are some users who are sensitive to max latency, such as stock exchange and online games.
You need to compare both at the same throughput to have any meaningful comparison.
I'm sorry for my lack of understanding, but could you tell me why you think so? When the user upgrades to 9.4 and runs the same workload, he would experience vastly increased max latency
The tests shown have not tested that. The test is not running the same workload on 9.4, but rather a vastly higher workload. If we were to throttle the workload in 9.4 (using pgbench's new -R, for example) to the same level it was in 9.3, we probably would not see the max latency increase. But that was not tested, so we don't know for sure.
--- or in other words, greater variance in response times. With my simple understanding, that sounds like a problem for response-sensitive users.
If you need the throughput provided by 9.4, then using 9.3 gets lower variance simply be refusing to do 80% of the assigned work. If you don't need the throughput provided by 9.4, then you probably have some natural throttling in place.
If you want a real-world like test, you might try to crank up the -c and -j to the limit in 9.3 in a vain effort to match 9.4's performance, and see what that does to max latency. (After all, that is what a naive web app is likely to do--continue to make more and more connections as requests come in faster than they can finish.)
