Your business and the heartbleed bug

Discussions
#1

Assuming you run some sort of server-based service that involves credentials (SaaS logins, license key checking, credit card payments, whatever), how have you reacted to the Heartbleed bug? Have you communicated to your users that you were affected/not affected? What kind of language did you use?

I’m struggling thinking of ways to explain the situation in a forthcoming way without causing panic or confusion. In my case my server software did not contain the vulnerability, but 3rd party services I use such as my credit card processing service were vulnerable. My audience is utterly non-technical. Nobody has asked me about it yet, but I’d rather be proactive.

I assume most of your businesses were affected in some way. How are you handling it?

#2

Twilio and IFTTT both actually sent me an email, so that puts them ahead of basically every other website I have an account with. (I think there was a third, I forget who.)

I’d love to see people hit the giant red “we may have been rooted” button, redeploy servers, and forcibly expire all user credentials. But I seriously doubt we’ll see much of that, even from e.g. banks.

I mean, crap, I got 2-3 emails tops.

I did actively contact my financial institutions, and I got boilerplate “our security team has been made aware of this bug” responses. Which I sort of expected. But so much as an email from them cautioning me to reset my passwords? Forced session & password expiration? Nada.

(P.S. if you have a bank / credit union that actually sent a “reset your passwords” email, and they’re usable from where I live, I’d love to open an account there. I’m not sure I’d care if was Bank of America at this point.)

#3

Been wondering what to do myself.

I’m on the MS stack so I’m not effected but should I send a email anyway?

Thinking that emailing will just raise other questions and I shouldn’t say anything?

#4

Not sending anything raises questions with customers who are aware of it.

We wanted to reassure you that your account with FooApp was not vulnerable to the Heartbleed bug.

Many of your other accounts around the web probably were, so it would be a good time to update your passwords and maybe start using a password vault like LastPass, which makes it easy to use strong unique passwords.

Also if users are reusing passwords as many, many users do, and if their email-password pair was compromised somewhere else…you may still see a spike in unauthorized account access. Breaches spread.

#5

I even don’t know how to handle this as a user!

I received some (not many) emails asking to change my password. And as I am aware of the bug, I tend to have a good impression of these services, not the opposite.

What bothers me is that I know we are supposed to change all our passwords, but how can I tell if a specific website has already patched? Should I change all my dozens of passwords and do it again in a few weeks?

Yesterday, my wife (she is a doctor) asked me about it and I didn’t know what to answer.

#6

Chromebleed is an extension for doing a Heartbleed test against whatever website you have open in Chrome. This runs a simple intrusion test and executes a limited version of the exploit, so it may be legally problematic in some jurisdictions.

By the same author, there’s also a web-based tester and a GitHub project which can be run locally as a command-line based tester (uses Go). The Chrome extension just wraps these and checks anything you visit.

Okay, the bad news: those won’t tell you if the site has refreshed their SSL keys. If Heartbleed was used to steal credentials from the site, and they don’t update the keys, then anyone who accessed them could pretend to be the site or access your HTTPS-protected communications.

For that, the best option I’ve seen yet is LastPass. When you use its security check feature, it now tells you the SSL cert status for any sites it’s storing a password for.

2 Likes
#7

Honesty is the best policy - the news (in the UK certainly) is full of scare stories and so called experts telling everyone the end of the world is nigh so most are aware something is going on and will appreciate your openness.

It is only a matter of time until people start asking about it and this way we can tell them our servers were “fixed” on the same day the exploit was announced, we communicated it to our users proactively and, as we have been totally open about it, we have nothing to hide so hopefully this will stop any issues arising.

1 Like
#8

For WiseCash, I first verified the situation, and was lucky to see the server itself wasn’t affected (like most people at EngineYard not using an ELB).

I felt that as a customer myself, I would have loved to be notified, so I sent this (inspired by Honeybadger):

Hi XXX,

last night, a severe OpenSSL vulnerability has been disclosed (HeartBleed).

Your WiseCash data is safe - we do not use a vulnerable version of OpenSSL.

Despite this, I’m in the process of renewing all our keys gradually, because your data safety matters to me.

Have a great day,

– Thibaut

PS: I’ll be adding an annual plan soon (one month free, only one transaction). Reply to this email if interested!

In retrospect (but well tough and tiring day, since I had to handle Heartbleed for my freelancing clients too), I should have phrased it better to be more explicit about the difference between the server and the third-parties I rely on too (which have access to a very little part of the data, but still).

I then made due diligence again, opened a google spreadsheet, and tracked down literally all the services I’m using (even remotely), writing down who had made a statement, in order to change the credentials /after/ they have fixed their systems. I still have a couple of keys to rotate on non critical bits, because not everyone has yet communicated. I also changed the Rails secret and reissued SSL certificates (even if not useful since the server isn’t vulnerable), mostly to train myself to handle this.

Later on, I explained what I did in a short blog post, which I linked from the sign-in form to make sure people would be aware.

A couple of customers replied, thanked me for the handling of the situation.

Some asked for the annual plan I proposed in PS too, which is nice :slight_smile:

Looks simple in retrospect, but I’ve been pretty concerned the whole time!

Hope this helps,

– Thibaut

#9

For my SaaS I did a lot of googling and twitter search to find out (for services which didn’t immediately send an email, which is most).

For “regular users”, check out this:

#10

Given the criticality + widespread coverage, I think a little email cannot hurt.

Plus, /you/ are on the MS stack, but what about your dependencies? (eg: DNS provider, email server etc etc?). It’s always good to think this through a bit, the ramifications are somewhat surprising.

#11

Yeah that’s the gotcha. I was all ready to gleefully send a “we’re not affected!” message out, until I realized Braintree Payment Solutions was in fact affected.

I think it’s bad to not send anything, but it’s worst to send something and then have to backpedal because you forgot about some 3rd party service. Then it looks like you don’t know what you’re doing.

Some 3rd party services don’t hold any identifiable data from users, so it’s not a problem if you don’t inform them specifically, but payment processors are the elephant in the room. They hold your user’s credit card info. I think informing them specifically about that situation is the right thing to do, even if the reality is nobody really knows how much info has leaked out.

Looks like Stripe was affected too by the way…

So after rotating my braintree API keys I posted a message explaining the situation and passing a link to Braintree’s report on the bug. No reaction yet.

#12

I dropped everything when Heartbleed hit, first verifying that we weren’t using a vulnerable version of the OpenSSL library on any of our servers (they’re… old), then spent a day upgrading lower-priority servers to the latest LTS release of Ubuntu so that if another bug drops we won’t be caught with our pants down.

I’ll do this on our high-priority servers (cough the product that pays the bills and has legal consequences for data disclosure cough) when I get a day free to do it right.

I didn’t send an Attn: All Customers email simply because we weren’t vulnerable and I think the average client of my products would be more scared by dodging a bullet than getting hit by one. That’s unfortunate, but that’s how non-technical people work. If we had been vulnerable, I would have sent it out immediately after remediation, emphasizing We Are Totally On Top Of This and Your Data Remains Safe.

#13

Our servers were vulnerable so I patched the servers, recompiled ruby, and reissued certs etc on the Tuesday the Heartbleed story broke (ironically, whilst away at a “Data Protection and Privacy” seminar!).

I struggled a bit about if/how to communicate this to our users though. We store HR performance plans & reviews so it’s pretty sensitive stuff (though thankfully not SSN, address, payment details etc) and our customers are local banks, small businesses etc. They really care about security (in a theoretical sense) but unfortunately know next-to-nothing about it (cough many are still on Windows XP). In the end, I waited a few days whilst I implemented some necessary features (like letting logged-in users actually change their passwords!) and sent the following email just to the admins for each company.

You may have heard of a recent vulnerability, “HeartBleed”, discovered in a popular encryption library used to secure Internet communications (the “https” in your browser). The flaw allowed an attacker to read parts of a server’s memory — potentially revealing user passwords, secret keys and other private information.

It’s estimated that over 60% of all sites on the internet were vulnerable to this flaw including Amazon, Google, Facebook, Yahoo, many online banks, government websites and, unfortunately, WorkCompass. The flaw was discovered early on Tuesday and we fixed our servers a few hours later once the upgraded packages became available for our operating system. Since then, we’ve also reissued our SSL certs, updated other parts of our software and reviewed our logs.

There is no evidence that WorkCompass was compromised at any time and there’s no evidence that your company data was accessed improperly.

At WorkCompass we take data security and privacy very seriously and respond quickly when issues like this arise. It’s also worth pointing out that our SSL configuration has the highest possible security rating and is stronger than most online banks, including those of AIB and Bank of Ireland.

However, out of an abundance of caution, if you want your employees to change their WorkCompass passwords they can do so here: or from the ‘Admin’ menu in the top-right hand corner.

If you have any further queries, don’t hesitate to contact me directly.

In hindsight I should probably have pushed them harder to change their passwords (because none have :frowning: ) but at the time I believed that the only people exploiting this bug were the NSA/GCHQ and I have no confidence that I (or, frankly, any small SaaS business) can defend against state-level actors.

1 Like