Today I learnt to sort of monitor my webserver live. And I learnt it the hard way… :-/
The LAMP servers kept crashing every 2 hours and I could not even log on with SSH. I had to hard reset them.
Here’s how I managed to watch them dying in real time. I opened an SSH window running
top -S -d 2 so I could see when the load average was going unusually high, what process was actually causing it. I was suspecting the IMAP server, but actually it was simpler than that: PHP & MySQL! :|
With a second SSH window (you HAVE to open these BEFORE things get bad or you just won’t be able to log in then) I ran a
tail acces_log -f on the Apache logs and I could see what requests were actually generating the high loads. 8|
It was a little tricky because those were not requests on huge PHP pages with lotsa SQL requests. No, those were rather plenty of requests on basic PHP pages with an average number of requests. It’s just that at some times there were too many of the same requests (on the same tables) at the same time. :o
Well, I’m not precisely a Unix guru, so there might actually be a more efficient way to find that kind of problems… but if you can’t find a better way, this one works.
Hum… something else I’d love to have would be a real time grapher that would show (in Windows) how the load average of several remote servers is evolving, and send out alerts if things get hot… Gotta look at that some day… ;)
Comments from long ago:
Comment from: Eike
try this: http://www.nagios.org then u can track every service running on ur box. And u will get a lot of notify mails, if u like ;-)
Comment from: Jason Edgecombe
check out Ganglia at http://ganglia.sourceforge.net/ It gives nice graphs of everything.
Comment from: Slamp
Nagios is great
Comment from: Sebastien
Hum… something else I’d love to have would be a real time grapher that would show (in Windows) how the load average of several remote servers is evolving, and send out alerts if things get hot…
This should be quite easy to achieve with SNMP and stuff like MRTG (ok, no alerts, but at least the graph feature is nice).
Comment from: Slamp
I think you can configure may be not MRTG but RRDTool to define some threshold alert
Comment from: Digitalxero