I've just been asked to troubleshoot a web server that had just started locking up for no apparent reason. Obviously the first thought was that it had been hacked ( some of the sites had, but no big deal - they got fixed on the fly ), so I logged in to have a look. Well, 30 minutes later I logged in, because that's how long it took to complete the process. It was a case of type some stuff and come back 10 minutes later. As the server was to all intents and purposes dead in the water (uptime eventually reported a load average of 170 for quad core machine), I used an one liner to kill the web server...

ps -ef | grep httpd | awk '{print $2}' | xargs kill -9

OK, look at the basics. Running through the standard checks on the hardware identified the problem - the CPU was spending 25% of it's time in waitio - and that's with no sites running. Looking for the culprit led me to the MySQL configuration - it's the standard one delivered with the OS! A bit of tweaking and here's the difference it made. Note the number of blocks read even though the blocks written is pretty constant - and this is the evil hours of the morning that I'm reporting stats for...

1. 2 nights ago

2. last night

And he load average is now comfortably below 1 again.