When good things break

posted by philip in Website Updates, Snooth

Over the last few days there have been several articles over how the technology blog, Mashable, has taken the number 1 spot from Techcrunch. Both sites now have approximately 2 million monthly readers, plus many more follow their news via email, twitter and other networks.

We’ve experienced first hand close to 10 times what an article from either of these giants can do to our daily traffic patterns. And, over the last few years we’ve added monitoring, cronitoring, redundancy, failover and other safety systems designed to handle these bursts. But last night, we screwed up.

Mashable wrote about our iPhone app, which, because it was brand new, and hosted on a different set of servers, did not have all the fail safes in place. As a result the index server that powers the search results for the iPhone crashed. We noticed the error at 6.50am Eastern time this morning, woke everyone up, and by 7.20am the server, and the service was back up. This affected the API server and the iPhone search results only.

We’re busy today adding the redundancy and failover solutions to the API and iPhone servers that keep the main Snooth.com site working smoothly. Apologies to anyone who was affected by the iPhone outage.

So far the feedback to the iPhone app has been great, one twitterer claiming that it made “an iPhone worth having”.

Here’s a sample of other chatter about the app:

UPDATE: The Mashable story is being highly Digg’d on Digg.com (a news site with 30M users per month). You can see the article and vote for it here. Please support Snooth and help us get to the home page.