Something interesting is going on. I wish I could show you the graphs,
but I'm sure this will not be a surprise to the seasoned veterans.
A particular application server I have has been running for over a
year now. I've been logging cpu load since mid-april.
It took 8 months or more to fall from excellent performance to
"acceptable." Then, over the course of about 5 weeks it fell from
"acceptable" to "so-so." Then, in the last four weeks it's gone from
"so-so" to alarming.
I've been working on this performance drop since Friday but it wasn't
until I replied to Arnau's post earlier today that I remembered I'd
been logging the server load. I grabbed the data and charted it in
Excel and to my surprise, the graph of the server's load average looks
kind of like the graph of y=x^2.
I've got to make a recomendation for a solution to the PHB and my
analysis is showing that as the dataset becomes larger, the amount of
time the disk spends seeking is increasing. This causes processes to
take longer to finish, which causes more processes to pile up, which
cuases processes to take longer to finish, which causes more processes
to pile up etc. It is this growing dataset that seems to be the source
of the sharp decrease in performance.
I knew this day would come, but I'm actually quite surprised that when
it came, there was little time between the warning and the grande
finale. I guess this message is being sent to the list to serve as a
warning to other data warehouse admins that when you reach your
capacity, the downward spiral happens rather quickly.
Crud... Outlook just froze while composing the PHB memo. I've been
working on that for an hour. What a bad day.
--
Matthew Nuzum
www.bearfruit.org
