What was your worst day running your SaaS? Share your experience!

Discussions
#1

Currently, I am having an issue with my SaaS platform - it’s a critical bug that affects the core functionality. Without this working correctly, my revenue is at stake.

The most disturbing is that I did not push any new code - this just suddenly stopped working.

This feels like a nightmare to me because I cannot find a reason why this doesn’t work and I am scared of those emails coming in soon with cancellation requests from my paying users :frowning:

So while I am struggling here, cheer me up a bit with your nightmare story running your service.

Oh dear, I wish this day could end already…

1 Like
#2

While it might seem like the end of the world, unless you’ve lost customer data, don’t worry too much about it.

Be honest with your customers about the issue. Let them know something broke in the software and that you are fixing it. You will find that customers can be pretty understanding of such issues, as long as you communicate effectively to them. They can also be quite accomodating of some serious downtime.

You are going to face issues like this every once in a while - the only surety in software is that “shit happens”. Everytime it will be a different issue you have not faced before - a bug that only is reproduced under specific conditions, a third party api that does not work as expected, a random server crash you cannot diagnose.

Data backup and restore strategy is your friend. Make sure to test backups for correctness.

My favorite personal story - after a straight 14 hour coding session, I inadvertently ran a mysql query on the production server that disabled all customer accounts and then I promptly went to bed. Woke up 12 hours later to a bunch of emails from paying users who were confused why they could not use the software anymore. Fixed data from backup, then replied to each of them with a sincere apology. Not one cancelation - infact not one angry reply.

2 Likes
#3

Not a SaaS, but way back when I was still selling SimplyTweet, my iPhone Twitter app and in the earlier days of Twitter when they did not have the streaming API, I had to poll the Twitter API to support push notifications for mentions and direct messages. It was all going well.

It was all good for months (or a year?). I was on my honeymoon with my wife. I checked my email and receive an email from Slicehost (remember them?) that I was generating too many DNS lookups and there were suspending my outgoing network traffic immediately. I asked for a few days grace, explaining my situation. I got a flat no. I can’t remember how I fixed it temporarily, but it was a very unhappy afternoon during one of the happier moments of my life.

That’s why you don’t want to check your email during your honeymoon.

1 Like
#4

I lost multiple hours worth of customer data. The full post-mortem. Here’s the lesson learned chapter of my book. And here’s the draft chapter of advice on how to handle big issues.

Long story short. Be honest. Be transparent. Treat your customers like you would want to be treated. It won’t be anywhere near as bad as you imagine it will.

1 Like