Hi all,
I’m trying to track down why some queries on my database system are intermittently much slower than usual. I have some queries that run, on average, 2-3ms, and they run at a rate of about 10-20 queries/second. However, every 3-5 seconds, one of the queries will be 500-100ms. This is making the average query time turn out to be closer to 20ms, with a very large standard deviation.
This happens to a number of otherwise very fast queries, and I’m trying to trace the reason. I’ve turned on lock logging and checkpoint logging, and this behavior happens whether or not a checkpoint is occurring. There are no lock waits happening in the system either.
Here’s my info:
- PostgreSQL 9.2.9 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4), 64-bit
- Installed from Postgresql Yum repo
- Changed Settings:
"application_name";"pgAdmin III - Query Tool";"client"
"archive_command";"/usr/crsinc/bin/wal_processor.sh %f %p";"configuration file"
"archive_mode";"on";"configuration file"
"autovacuum_analyze_scale_factor";"0.1";"configuration file"
"autovacuum_analyze_threshold";"500";"configuration file"
"autovacuum_naptime";"10min";"configuration file"
"autovacuum_vacuum_cost_delay";"50ms";"configuration file"
"autovacuum_vacuum_cost_limit";"250";"configuration file"
"autovacuum_vacuum_scale_factor";"0.1";"configuration file"
"autovacuum_vacuum_threshold";"1000";"configuration file"
"bgwriter_delay";"100ms";"configuration file"
"bgwriter_lru_maxpages";"1000";"configuration file"
"bytea_output";"escape";"session"
"checkpoint_completion_target";"0.7";"configuration file"
"checkpoint_segments";"128";"configuration file"
"checkpoint_timeout";"30min";"configuration file"
"checkpoint_warning";"895s";"configuration file"
"client_encoding";"UNICODE";"session"
"client_min_messages";"notice";"session"
"cpu_tuple_cost";"0.001";"configuration file"
"DateStyle";"ISO, MDY";"session"
"default_text_search_config";"pg_catalog.english";"configuration file"
"effective_cache_size";"94GB";"configuration file"
"effective_io_concurrency";"100";"configuration file"
"hot_standby";"off";"configuration file"
"hot_standby_feedback";"off";"configuration file"
"lc_messages";"en_US.UTF-8";"configuration file"
"lc_monetary";"en_US.UTF-8";"configuration file"
"lc_numeric";"en_US.UTF-8";"configuration file"
"lc_time";"en_US.UTF-8";"configuration file"
"listen_addresses";"*";"configuration file"
"log_checkpoints";"on";"configuration file"
"log_destination";"stderr";"configuration file"
"log_directory";"pg_log";"configuration file"
"log_filename";"postgresql-%a.log";"configuration file"
"log_line_prefix";"< user=%u db=%d host=%h time=%t pid=%p xid=%x>";"configuration file"
"log_lock_waits";"on";"configuration file"
"log_min_duration_statement";"100ms";"configuration file"
"log_rotation_age";"1d";"configuration file"
"log_rotation_size";"0";"configuration file"
"log_temp_files";"0";"configuration file"
"log_truncate_on_rotation";"on";"configuration file"
"logging_collector";"on";"configuration file"
"maintenance_work_mem";"6047MB";"configuration file"
"max_connections";"1300";"configuration file"
"max_stack_depth";"2MB";"environment variable"
"max_wal_senders";"4";"configuration file"
"port";"5432";"command line"
"random_page_cost";"1.9";"configuration file"
"shared_buffers";"8GB";"configuration file"
"ssl";"on";"configuration file"
"ssl_renegotiation_limit";"0";"configuration file"
"standard_conforming_strings";"off";"configuration file"
"superuser_reserved_connections";"10";"configuration file"
"TimeZone";"US/Pacific";"configuration file"
"wal_keep_segments";"64";"configuration file"
"wal_level";"hot_standby";"configuration file"
"work_mem";"188MB";"configuration file"
OS: CentOS release 6.7 (Linux databasep1.crsinc.com 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux). Our server has 24 cores and 128 GB of RAM.
This happens via psql, JDBC, PDO connections.
Thanks in advance for any help!
Scott
This email message contains information that Motus, LLC considers confidential and/or proprietary, or may later designate as confidential and proprietary. It is intended only for use of the individual or entity named above and should not be forwarded to any other persons or entities without the express consent of Motus, LLC, nor should it be used for any purpose other than in the course of any potential or actual business relationship with Motus, LLC. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately and destroy the original message.
Internal Revenue Service regulations require that certain types of written advice include a disclaimer. To the extent the preceding message contains advice relating to a Federal tax issue, unless expressly stated otherwise the advice is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer, for the purpose of avoiding Federal tax penalties, and was not written to support the promotion or marketing of any transaction or matter discussed herein.
