What’s New in the Amazon Cloud?: Security Vulnerability in Amazon EC2 and SimpleDB Fixed (7.5 Months After Notification)
By Craig Balding
Recent news stories highlight new features and capabilities, including a SQL-like SELECT API for Amazon SimpleDB and the (significant news) that Cloud Compute is now available in the EU.
More worryingly, however and missing from the Amazon AWS homepage, is that Amazon just rolled out an important security fix to correct a basic, but significant, cryptographic weakness in their request signing code. The flaw affects their database API (SimpleDB) and compute API (EC2) services.
Colin Percival discovered the weakness and reported it to Amazon back in May (!). Thats 7.5 months ago. Here’s the “executive summary” from Colins’ write-up:
AWS signature version 1 is insecure
The important bit first: If you are making Query (aka REST) requests to Amazon SimpleDB, to Amazon Elastic Compute Cloud (EC2), or to Amazon Simple Queue Service (SQS) over HTTP, and there is any way for an attacker to provide you with data which you use to construct your request, switch to HTTPS or start using AWS signature version 2 now. For example, if you allow users to add arbitrary “tags” to documents, and you use SimpleDB to store those tags, this means you. (Amazon Flexible Payments Service (FPS) and Amazon Devpay also use the same insecure signature method, but they already require the use of HTTPS. Amazon S3 and other services use different signature methods.)
So in a nutshell, if you only ever accessed SimpleDB or EC2 services over HTTPS, you’re not impacted.
But if you did use the non-encrypted transport, you were vulnerable to a man-in-the-middle attack (MITM) that could expose your data hosted in SimpleDB and your EC2 compute instances.
Overall, Colin is generous in his praise of the response from the Amazon team - although did express concern at the time it took for the vulnerably to be closed:
I reported this issue to Amazon via an email to Jeff Barr, the “Lead Web Services Evangelist” at Amazon on May 1st, and while it took a long time — 7.5 months — for it to be fixed, I’m happy to say that Amazon took this issue seriously at all times, and the lengthy timeline was simply because of the large amount of work involved. Jeff forwarded my email to someone working on SimpleDB (I’ve been asked not to mention names), who confirmed that they agreed that this was a problem. As part of their review of my findings, Amazon’s security people realized that this also affected EC2 and SQS — in my initial investigation I had only looked at SimpleDB — and at the beginning of July they agreed to send me their planned signature version 2 so that I could review it.
Based on the timeline provided by Colin, it took about 2 months for Amazon to assess the extent of the exposure across their other service APIs and come up with a proposed fix. That’s a considerable length of time given the simplicity of the weakness. They then took over 5 months to actually implement the fix. I don’t care how many client libraries you offer Mr Cloud Provider, thats still a very, very long time to leave your customers exposed while you get a fix into production.
Back to Colin as he describes the vulnerable signing process:
AWS signature version 1 signs an HTTP query string as follows:
- Split the query string based on ‘&’ and ‘=’ characters into a series of key-value pairs.
- Sort the pairs based on the keys.
- Append the keys and values together, in order, to construct one big string (key1 + value1 + key2 + value2 + … ).
- Sign that string using HMAC-SHA1 and your secret access key.
Did you spot the problem? Colin did and explains as follows:
When Amazon invented this signature scheme, they forgot about one of the foremost design principles relating to cryptographic signatures: Collisions are BAD! In a well-designed signature system, it should be computationally infeasible to construct two different messages which have the same signature; this prevents substitution attacks where an attacker convinces the key holder to sign a “harmless” message, and then attaches that signature to a different message. Looking at how AWS signature version 1 is computed, it’s easy to see how to construct collisions: Because there are no delimiters between the keys and values, the signature for “foo=bar” is identical to the signature for “foob=ar”; moreover, the signature for “foo=bar&fooble=baz” is the same as the signature for “foo=barfooblebaz”.
Oops. He then goes on to explain how a man in the middle attack could exploit the weakness.
Taking a step back, what are the concerns with this weakness and the way Amazon handles security vulnerability reports in general?
Vulnerability Reporting
- Amazon doesn’t provide a visible mechanism for researchers/customers to report security vulnerabilities. You can report ‘abuse’ (i.e. your EC2 image is under attack) via a contact form on the AWS website but unless you know someone at Amazon, you have to guess how to report a security vulnerability.
- Amazon has no published policy on how they handle vulnerability reports. Nothing to explain their commitment to following up, promptly responding to reports, overall process etc. Now, it seems they do follow up and overall Colin says good things but that timeline sends shivers up my spine.
- Amazon doesn’t have a dedicated security page, mailing list, RSS feed or pigeon carrier for alerting customers of security issues. This makes it a PITA for enlightended customers to even know they are at risk (unless they setup Google News Alerts) and comes across to me at least as a trifle arrogant.
Vulnerability Notification
- Amazon hasn’t notified their customers yet that their SimpleDB and EC2 instances were vulnerable for the past 7.5 months (I’m a customer and I’ve yet to receive an email). Obviously the vulnerability itself goes back further than 7.5 months.
- Amazon have not published how customers can determine if the issue was exploited (back to the visibility issue again). Clearly, Amazon doesn’t have full visibliity of the network between the customer and their data centers, however what analysis has Amazon performed on the logs *only they have access to* that could highlight an issue? (frankly, due to the nature of this issue and the lack of event logs/audit trails I suspect the simple answer is there isn’t much they can do here).
Secure Software Development
- Amazons’ internal code review process failed to detect a basic cryptography flaw. Note, although the error is pretty basic, errors do happen. The real problem here is the failure of peer review processes to identify the problem prior to going into production. Cryptography routines along with other sensitive security routines should be subject to greater security analysis as clearly the stakes are higher. If they did in fact have this code reviewed for security weaknesses, this is a huge oversight. So this is now fixed, what else is vulnerable? It would be great to learn that the lengthy delay in delivering the fix was due to Amazon have all their source code reviewed for security issues…
- Amazon have made no public commitment regarding how security is factored into their software development lifecycle. Until now they could argue they didn’t need to - they hadn’t messed up (or at least, we don’t think they did).
The geek in me loves Amazon AWS and the promise that it holds, but the Security Professional rooted in the “here and now” is seeing too many red flags (the issues are wider than the handling of this one bug).
If anyone from Amazon reads this and wants to talk security roadmap and go “beyond the whitepaper”, you know where I am.