Thread: High load average every 105 minutes
I have Postgresql 9.3 installed on AWS instance ubuntu 14.04
The load average of the instance increase every 105 minutes even without any database. I’ve checked the scheduled jobs but couldn’t find anything suspicious. When the load average was at peak, I couldn’t see any process consuming resources. This happens on all my servers that has postgresql installed, even after I purge postgresql. Has anyone seen this before?
Here’s my config:
postgresql.conf
data_directory = '/var/lib/postgresql/9.3/main'
hba_file = '/etc/postgresql/9.3/main/pg_hba.conf'
ident_file = '/etc/postgresql/9.3/main/pg_ident.conf'
external_pid_file = '/var/run/postgresql/9.3-main.pid'
listen_addresses = '*'
port = 5432
max_connections = 100
unix_socket_directories = '/var/run/postgresql'
ssl = true
ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem'
ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key'
shared_buffers = 128MB
shared_preload_libraries = 'pg_stat_statements'
wal_level = hot_standby
checkpoint_segments = 8
max_wal_senders = 3
wal_keep_segments = 8
log_min_duration_statement = 300
log_line_prefix = '%m '
log_timezone = 'UTC'
datestyle = 'iso, mdy'
timezone = 'UTC'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
default_text_search_config = 'pg_catalog.english'
Nhan Nguyen
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
On 11/4/2016 1:45 AM, Nhan Nguyen wrote: > > The load average of the instance increase every 105 minutes even > without any database. I’ve checked the scheduled jobs but couldn’t > find anything suspicious. When the load average was at peak, I > couldn’t see any process consuming resources. This happens on all my > servers that has postgresql installed, even after I purge postgresql. > Has anyone seen this before load average doesn't mean much other than the number of processes waiting for a resource. does the IO latency spike at this time, too? with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is pretty much out of your control. -- john r pierce, recycling bits in santa cruz
Thanks for the reply.
I found no strange processes or queries while load average was at peak. IO also didn't change. Some more slow queries were logged, but not many.I think sharing the VM with other customers doesn’t have much to do with this. I checked my other servers too, and only those that have postgresql have the load average issue. Generally it doesn’t impact my system much, but when there are slow queries, this issue just makes everything worse.
Nhan Nguyen
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
On Nov 4, 2016, at 4:03 PM, John R Pierce <pierce@hogranch.com> wrote:On 11/4/2016 1:45 AM, Nhan Nguyen wrote:
The load average of the instance increase every 105 minutes even without any database. I’ve checked the scheduled jobs but couldn’t find anything suspicious. When the load average was at peak, I couldn’t see any process consuming resources. This happens on all my servers that has postgresql installed, even after I purge postgresql. Has anyone seen this before
load average doesn't mean much other than the number of processes waiting for a resource. does the IO latency spike at this time, too?
with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is pretty much out of your control.
--
john r pierce, recycling bits in santa cruz
--
Sent via pgsql-general mailing list (pgsql-general@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general
>> with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is prettymuch out of your control. > I found no strange processes or queries while load average was at peak. IO also didn't change. Some more slow queries werelogged, but not many. > I think sharing the VM with other customers doesn’t have much to do with this. I checked my other servers too, and onlythose that have postgresql have the load average issue. Generally it doesn’t impact my system much, but when there areslow queries, this issue just makes everything worse. Hi, generally speaking AWS is pretty good at isolating users (and you can request single tenancy machines or dedicated machines as well if you're concerned about this). However, if you're running t1 or t2 instances, you get the concept of CPU credits. When those run out, your system is slowed down until the credits recover. I could imagine that this way some cyclic load patterns emerge, if there is constant load on the machines. Nhan, what instance types are you running? Bye, Chris.
>> with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is prettymuch out of your control. > I found no strange processes or queries while load average was at peak. IO also didn't change. Some more slow queries werelogged, but not many. > I think sharing the VM with other customers doesn’t have much to do with this. I checked my other servers too, and onlythose that have postgresql have the load average issue. Generally it doesn’t impact my system much, but when there areslow queries, this issue just makes everything worse. Hi, generally speaking AWS is pretty good at isolating users (and you can request single tenancy machines or dedicated machines as well if you're concerned about this). However, if you're running t1 or t2 instances, you get the concept of CPU credits. When those run out, your system is slowed down until the credits recover. I could imagine that this way some cyclic load patterns emerge, if there is constant load on the machines. Nhan, what instance types are you running? Bye, Chris.
>> with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is prettymuch out of your control. > I found no strange processes or queries while load average was at peak. IO also didn't change. Some more slow queries werelogged, but not many. > I think sharing the VM with other customers doesn’t have much to do with this. I checked my other servers too, and onlythose that have postgresql have the load average issue. Generally it doesn’t impact my system much, but when there areslow queries, this issue just makes everything worse. Hi, generally speaking AWS is pretty good at isolating users (and you can request single tenancy machines or dedicated machines as well if you're concerned about this). However, if you're running t1 or t2 instances, you get the concept of CPU credits. When those run out, your system is slowed down until the credits recover. I could imagine that this way some cyclic load patterns emerge, if there is constant load on the machines. Nhan, what instance types are you running? Bye, Chris.
I forgot to mention, my DB is running on a c4.xlarge instance.
Nhan Nguyen
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
System Engineer
MB: (+84) 934 008 031
Skype: live:ducnhan813
On Nov 7, 2016, at 5:03 PM, Chris Mair <chris@1006.org> wrote:with AWS, your system is sharing the vendors virtual machine environment with other customers, and performance is pretty much out of your control.I found no strange processes or queries while load average was at peak. IO also didn't change. Some more slow queries were logged, but not many.
I think sharing the VM with other customers doesn’t have much to do with this. I checked my other servers too, and only those that have postgresql have the load average issue. Generally it doesn’t impact my system much, but when there are slow queries, this issue just makes everything worse.
Hi,
generally speaking AWS is pretty good at isolating users (and you can request single tenancy machines or
dedicated machines as well if you're concerned about this).
However, if you're running t1 or t2 instances, you get the concept of CPU credits. When those run out, your
system is slowed down until the credits recover. I could imagine that this way some cyclic load patterns
emerge, if there is constant load on the machines.
Nhan, what instance types are you running?
Bye,
Chris.