Amazon Web Services gives developers access to massive computing capability. Now hackers have found ways to hijack some accounts and use that power to make money on someone else’s dime.
Joe Moreno’s bill for Amazon Web Services is usually about $5 a month. But last Thursday, he learned his AWS credentials had been compromised. An unknown person started renting computing power from Amazon on his account, racking up more than $5,300 in charges on servers in Amazon data centers as far away as Tokyo, São Paulo, Sydney, and Singapore.
It appeared that he was running processes that “mined” Bitcoin—creating units of the digital currency in exchange for processing transactions.
We Have Met The Enemy, And He Is Us
Given the timing of the attack, Moreno initially thought the Heartbleed bug was to blame, until he tracked down the breach and realized it was his own error.
In addition to developers’ usernames and passwords for their accounts, AWS uses “access keys” which are easier to include in software. And that’s the problem—developers include them in software, including copies of the software they store in public source-code repositories like GitHub.
Moreno had uploaded code to a GitHub repository, inadvertently including his Amazon credentials.
You might think this is an isolated case, but a security expert in Australia discovered almost 10,000 AWS credentials in a search of GitHub last month.
Ty Miller, founder of security testing firm Threat Intelligence, found exposed credentials for Amazon, Google’s Cloud Platform, and Microsoft Azure in GitHub repositories, but the largest number were for Amazon.
“These credentials are likely to provide full access to the AWS account,” Miller told ReadWrite. That means hackers could delete data or add data and start new computing processes which could perform just about any task.
Amazon appears to be aware of the problem. The company specifically warns developers against including their credentials in code that they upload. But it’s not clear how Amazon can police the problem.
Amazon For Nothing And Your Servers For Free
Moreno discovered the breach to his account after receiving email from Amazon asking him to update his credit-card information. Moreno, a former software developer at Apple, logged in and noticed the charges. He immediately contacted Amazon.
“Your AWS credentials have been compromised,” the Amazon representative said. Bitcoin mining was a common goal of these hackers, though the AWS computing resources could be used for all kinds of money-making schemes.
When software consultant Ted Howard learned of Moreno’s experience, he commiserated. On April 5, he had learned that his Amazon account had been hacked.
“I immediately changed my password, disabled my access key and created a new key,” he told ReadWrite.
Howard also believes the breach was likely his fault. After checking his GitHub repository, he found that he had committed a file that contained his AWS access key.
“I seem to be incapable of escaping my own stupidity,” he said. But the unintentional publication of AWS credentials appears to be a common problem. It even happened to security researcher Rich Mogull in January.
Keys To The Computing Kingdom
Howard thought his immediate problem was over, though he still had the bill to settle with Amazon.
But on Friday, after communicating with Moreno, he discovered yet another security breach on his AWS account, despite the steps he had already taken to secure it.
After Moreno’s Amazon troubles came to light, Howard logged back into his own Amazon account and saw that 13 new EC2 instances in Oregon had been started—on April 9, days after he learned of $6,000 in fraudulent charges on his account.
“Of course I changed my password and disabled my new access key,” he said. “This time I didn’t even bother creating a new one.”
Since he hadn’t used the new access key anywhere, or uploaded or shared it anywhere, he was worried.
“Whether it’s related to Heartbleed is anyone’s guess,” Howard said. “It’s possible that they still accepted requests with the old access key after I killed it. Perhaps the attacker was notified of the new key somehow. I really have no idea.”
Amazon: “We’ve Been Seeing More And More Of This”
Later on Friday, Amazon told Howard that the hacker may have used a feature called “Spot Requests” on his account before he reset his credentials. He checked out his account and found many of them.
As an Amazon developer, you can bid on unused computing resources via Spot Requests, and when Amazon accepts the price you set, it automatically starts using the designated computing resources. Amazon told Howard he had to check each of Amazon’s geographical region for such requests, as deleting one would not affect instances in any other region.
“The nefarious way to use this is to set up a ton of requests with a high max price,” Howard said. “Even once all the credentials are changed, this request is still present, so new instances continue to be spun up and down over time. This is apparently what happened to me.”
That’s what an Amazon representative told Moreno the day he discovered the breach. The Amazon employee also told Moreno to check his EC2 spot instances in other regions, and predicted he would see high end instances running. Which he did.
Like Howard, Moreno changed his password, but took the extra precaution of removing his code from GitHub. That’s not a trivial process: Because the way repositories are backed up, his old keys may still be discoverable.
A helpful GitHub tutorial explains how to purge files from your repositories permanently and avoid accidental commits in the future.
Plugging The Holes
Recently, Amazon has changed the way it generates credentials, Moreno and Howard both said. To allow programs to access AWS resources, you used to need an access key ID and a secret access key—strings of characters generated by Amazon. In the past, you could log into your account and retrieve the secret key at any time. That’s no longer the case.
“If you lose [the secret key], you must disable and generate a new access key,” Howard said.
An Amazon guide for managing AWS credentials suggests removing, or not generating, an access key for your root account; and using AWS Identity Access Management (IAM) to create temporary security credentials for applications that interact with AWS resources. It also explains how to manage IAM access keys.
“We take security very seriously at AWS, and we provide many resources, guidelines and mechanisms to help customers configure AWS services and develop applications using security best practices,” an AWS spokesperson said. “When we become aware of potentially exposed credentials, we proactively notify the affected customers and provide guidance on how to secure their access keys.”
It seems that Amazon could do more, however. If security researchers can easily scan public sites like GitHub and find access keys, couldn’t Amazon do the same, and save itself and its customers from these incidents by immediately deactivating the keys?
How To Protect Yourself
It may go without saying, but if you’ve uploaded code to GitHub, you might want to check whether you inadvertently included your credentials for anyone, including hackers, to access.
“I’m sure many developers have made the mistake I’ve made,” Moreno said. He and Howard offer the following advice.
- Use two-factor authentication. Although this would not have helped either Howard or Moreno, additional security through a second type of authentication helps protect email and other accounts which might also hold your cloud keys. Take advantage of it.
- Never hardcode your cloud computing credentials. Even if you’re using a private source-code repository, that may change in the future. You may decide to contribute code to an open-source project, for example. “After looking through my code, I see that I had hardcoded my credentials and then commented out that code, later, when I moved the credentials to a preferences file,” Moreno said. But, even that isn’t good enough since preferences files are usually checked into repositories with code.
- Use Identity Access Management. This feature from Amazon lets you create individual accounts that have limited privileges. If you wanted to create an applications that stores its data in S3, you can create an account that only has access to one S3 bucket. “If that app got compromised or those credentials got accidentally checked in to GitHub, then only that particular S3 bucket would be exposed,” Howard said.
And if that doesn’t stop a hack, you’ll still want to gather information about what happened. Mogull explained in a post how to take a snapshot and apply forensic analysis to a hack.
The most important advice Howard offers?
“Don’t put your Amazon credentials into source code and then share that source code in a public place like GitHub!”
It seems obvious. But it’s clear that thousands of developers haven’t taken this obvious step.
Update: After we published this story, Joe Moreno received this email from Amazon.
[email protected] wrote:
Hello Joseph,
I have good news! As a one-time exception, we’ve approved a credit of the charges for April for the amount of $5360.23 This credit will offset the amount of your compromised resources!
Please make sure to monitor your AWS usage periodically to avoid unexpected charges. By selecting Bills in your Account Billing console, you can see current and past usage activity by service and region.
Photo courtesy of Joe Moreno