Home Exterminating Heartbleed: How To Clear It Out Of Your Data Center

Exterminating Heartbleed: How To Clear It Out Of Your Data Center

If your house has ever been infested with roaches, you know that for every one you see, there are many more lurking hidden. The Heartbleed bug is the same way.

See also: What You Need To Know About Heartbleed

It’s not just in Web servers. It may be hidden deep throughout your organization in virtual private networks (VPNs), routers, virtual machines, disk arrays, server management software, mail servers, and phones. To completely protect users of your apps or services against Heartbleed, you need to conduct a full extermination sweep—which, by the way, involves a lot more than just patching the buggy version of OpenSSL, the cryptographic security software where the vulnerability surfaced.

Of course, you should start with OpenSSL updates. If you don’t update, Heartbleed might allows hackers to scroll through the system memory on affected devices and snatch private encryption keys and passwords to do things like steal taxpayer IDsimpersonate virtual private networks, and even bypass multi-factor authentication in VPNs. Yikes.

In short, your to-do list looks like this:

  • Identify vulnerable systems throughout your IT infrastructure
  • Update OpenSSL on each system
  • Generate new public-private encryption key pairs for each system
  • Generate new SSL certificates that verify servers and other systems
  • Revoke your old SSL certificates

See also: After Heartbleed, “Forward Secrecy” Is More Important Than Ever

That list may sound straightforward, but there are still a lot of unknowns, particularly in that first step. “What level of paranoia, or prudence, do you want to follow?” asks Christopher Aker, CEO of cloud hosting provider Linode. “Because OpenSSL is statically compiled into so many applications, you need to figure out what you’re using that may contain it.”

How SSL Works And Why You Need To Update

OpenSSL is an open-source implementation of the HTTPS security protocol that provides a secure, private channel between, say, your browser and a secure server. Establishing that connection is a little bit like making a phone call on a line that can easily be tapped. First, how can you know for certain the person on the other side is who they say they are? And once you’re sure, how do you make sure no one else can listen in on your conversation?

To make a secure Internet connection, a person must connect to a website with a valid certificate, typically one issued by a known and trusted certificate authority. The certificate is signed with a public encryption key that effectively identifies its rightful owner—in this case, the organization running the server on one hand, and the browser maker on the other. Any certificate owner also holds a matching private encryption key it can use to prove ownership of the certificate.

See also: Understanding Encryption—Here’s The Key

(In practice, it’s a little more complicated than that; see this Wikipedia example for more detail. Normally, once the server has authenticated its identity to the browser, they securely exchange yet another temporary encryption key and use that to secure the remainder of their “conversation.”)

Certificate authorities like Symantec, Digicert, Comodo, and even GoDaddy generate, sign, and manage certificates, and you interact with them when setting up a Web server, or installing software that uses SSL for deeper security. Websites are the most common applications for OpenSSL is used, but mail servers, chat servers, and other applications use OpenSSL as well. Large organizations sometimes run their own certificate authorities or self-sign certificates to allow their devices to communicate securely with one another without involving a certificate authority.

All well and good. Heartbleed, however, can expose information stored in server memory—which sometimes can include the private key associated with its certificate. Once an attacker has a private key in hand, it’s not that difficult to create ersatz certificates and set up copycat sites that can lure in users and steal their personal information. The same private key can also be used to decrypt secure traffic—quite possibly even historical traffic, if someone’s been recording it for a while.

If you’re using one of the affected versions of the OpenSSL library (OpenSSL version 1.0.1 up to and including revision f, and 1.0.2-beta1), make sure to update to the fixed version—1.0.1g, and reboot your server to make sure it isn’t using an old version in memory. (Same goes for any servers you’re running virtually in cloud “instances.”) Ubuntu.org has a guide; Apachefriends has another

Re-Certify Yourself

You’re not done yet. To be safe, you need to get new SSL certificates. “With Heartbleed, fixing it is only a small portion of what needs to be done in the grand scene of things,” said Will Dormann, a member of the vulnerability team in the CERT division at Carnegie Mellon University’s Software Engineering Institute. “After that, there’s a bit of cleanup.”

Updating your certificates is not such a big deal, it turns out. Some certificate authorities, like Symantec and GoDaddy, are allowing you to do so for free. As part of that, you’ll need to generate a new private key and matching public key for each new certificate.

Many vendors, including Google, encourage exactly that. “In light of new research on extracting keys using the Heartbleed bug, we are recommending that Google Compute Engine (GCE) customers create new keys for any affected SSL services,” Google wrote earlier this month.

CyberGhost, the maker of virtual private network software that lets you travel anonymously across the web, finished replacing all of the SSL certificates on all its more than 300 servers the week before last. Timo Beyel, CTO of CyberGhost, said it would have been easy to take all the servers down and update them all using a script but he chose to take them down a few at a time to minimize service disruption to customers. 

“For us, it was very clear, that there might be possible vulnerability,” Beyal said. “That’s why we updated all our VPN servers, without waiting for the actual proof of concept.” Which came soon afterward, when the Swedish secure-communications firm Mullvad showed it was possible to extract private keys from an OpenVPN server that relied on OpenSSL.

It’s better to play it safe and generate a new certificate for each server, according to CERT’s Dormann. Assuming or hoping that your site wasn’t hacked before you became aware of Heartbleed, and not responding, is a risky decision. “The safest thing is to assume the worst, and figure out what to do,” he said.

Trash Those Old Certificates

Some companies are focusing on generating new certificates and neglecting the necessary step of revoking the old certificates. If someone accessed a site before it obtained a new certificate—and there was plenty of time to do that, since the bug has been in the wild for more than two years—a hacker could potentially impersonate the server if the old certificate hasn’t been revoked. And unrevoked certificates can be valid for years.

But check with your certificate authority on how they handle revocations. GoDaddy, for instance, advises not to revoke the certificate yourself, or you may find yourself in a nebulous situation:

NOTE: We automatically deactivate the previous certificate when we issue the new, re-keyed certificate. Do not revoke unless you are certain you want to cancel the existing certificate. When you revoke, the SSL credit is canceled and you cannot re-key the certificate.

Cancelling and reissuing privacy certificates is “a fast process,” said Shawn Marck, chief security officer at the attack-mitigation firm Black Lotus. “The main concern is that mass revocation of SSL certificates will cause strain on CA infrastructure.”

So far, that hasn’t been the case. On the other hand, certificate revocations spiked in mid-April but then quickly flatlined again, which suggests that the certificate system hasn’t yet had to cope with a true mass revocation.

Where Else Should You Look?

Like roaches, buggy OpenSSL instances may be hiding in the darker corners and less obvious areas of your network. Contact all your vendors to find out whether products you use are vulnerable to Heartbleed exploits. CERT has a handy vulnerability note with a growing list of vendors that are affected.

Hint: Take a close look at server virtualization, virtual desktop infrastructure, networking and security. If you use any VMware products, you might be in for some overtime. VMware released patches for all its affected products, including its hypervisor, ESXi 5.5, vCenter, and VDI clients. VMware advises:

  1. Deploy the VMware product update or product patches
  2. Replace certificates per the product-specific documentation
  3. Reset passwords per the product-specific documentation

Microsoft’s virtualization hypervisor, Hyper-V, isn’t vulnerable to Heartbleed, but some Citrix VDI clients are.

Cisco advises replacing and revoking certificates at the router, phone, and VPN level once devices are patched. “Out of an abundance of caution, we recommend revoking the certificates for devices (Cisco and non-Cisco) once a patch has been applied,” a spokesperson said. Cisco outlines how to revoke certificates in product configuration guides like this one for the Cisco VCS Expressway video server.

If your vendor isn’t forthcoming with information about vulnerabilities or fixes, you can use a tool like CrowdStrike’s Heartbleed Scanner to look in the cracks. It claims to scan “Intranet websites, OpenSSL VPNs, secure FTP servers, databases, secure SMTP/POP/IMAP email servers, routers, printers, phones, and anything else that may have been compiled with OpenSSL 1.0.1-1.0.1f.”

Are you experiencing Heartbleed? Let us know in the comments. 

Lead image by Flickr user Zhao!; certifcates image by Flickr user Dennis Wong; Seattle city trash-can image courtesy of Flickr user Seattle Municipal Archives

About ReadWrite’s Editorial Process

The ReadWrite Editorial policy involves closely monitoring the tech industry for major developments, new product launches, AI breakthroughs, video game releases and other newsworthy events. Editors assign relevant stories to staff writers or freelance contributors with expertise in each particular topic area. Before publication, articles go through a rigorous round of editing for accuracy, clarity, and to ensure adherence to ReadWrite's style guidelines.

Get the biggest tech headlines of the day delivered to your inbox

    By signing up, you agree to our Terms and Privacy Policy. Unsubscribe anytime.

    Tech News

    Explore the latest in tech with our Tech News. We cut through the noise for concise, relevant updates, keeping you informed about the rapidly evolving tech landscape with curated content that separates signal from noise.

    In-Depth Tech Stories

    Explore tech impact in In-Depth Stories. Narrative data journalism offers comprehensive analyses, revealing stories behind data. Understand industry trends for a deeper perspective on tech's intricate relationships with society.

    Expert Reviews

    Empower decisions with Expert Reviews, merging industry expertise and insightful analysis. Delve into tech intricacies, get the best deals, and stay ahead with our trustworthy guide to navigating the ever-changing tech market.