One of my sites was rebooted by the hosting company this weekend. The external monitor (site24×7) dutifully reported that it went down, and then that it was back up. Seeing that it had come back up, I relaxed.

However, in checking the logs last evening, I found that there were 503 errors occurring, and upon checking, some key site pages were down!

Why did my external monitor not detect this? I had it checking for keywords that were coming from the database, or so I thought.

It was a simple oversight. The page I was checking was cached! So although it appeared that the monitor was getting a page from Rails, I was really just getting a cached page from Apache, and Mongrel was dead.

Moral of the story: make sure your external monitors are not only checking for data that comes from the database, but also that they are hitting pages that are not cached.