Cloud Security

Put simply, these are virtual machine instances that you can pay to have reserved for you to use anytime you want. The benefit should be assured availability and lower cost for machines with heavy uptime requirements.  You still only pay for what you use, in addition to a one time non-refundable payment to make the reservation.

When I first read about this I got quite excited.  I was already thinking of the potential use cases from a security perspective, particularly around business continuance/disaster recovery.

In fact, this is something fellow AWS customers have been asking for:

Also, quite a few customers actually told us something even more interesting: they were interested in using EC2 but needed to make sure that we would have a substantial number of instances available to them at any time in order for them to use EC2 in a DR (Disaster Recovery) scenario. In a scenario like this, you can’t simply hope that your facility has sufficient capacity to accommodate your spot needs; you need to secure a firm resource commitment ahead of time.

This is where it gets interesting.  What does ‘firm resource commitment ahead of time’ mean?

Always Read The Label

I surfed over to the AWS Customer Agreement to read the small print:

8.3. Reserved Instance Pricing. You may designate EC2 instances as subject to the reserved pricing and payment terms (“Reserved Instance Pricing”) set forth on the EC2 detail page on the AWS Website (each designated instance, a “Reserved Instance”). You may designate instances as Reserved Instances solely by calling to the Purchasing API (the “API Call”). In the API Call you must designate an availability zone, instance type and quantity for the applicable Reserved Instances. The Reserved Instances may only be used in the designated availability zone. We may change Reserved Instance Pricing at any time but price changes will not apply to previously designated Reserved Instances. We may terminate the Reserved Instance Pricing program at any time. Notwithstanding anything to the contrary herein, Reserved Instances are nontransferable and all amounts paid in connection with the Reserved Instances are nonrefundable, except that if we terminate this Agreement pursuant to Section 3.3 or terminate the Reserved Instance Pricing program we will refund you a pro rata portion of any up-front fee paid in connection with any previously designated Reserved Instances. In addition to being subject to Reserved Instance Pricing, Reserved Instances are subject to all data transfer and other fees applicable under this Agreement.

Hmm, suddenly I lost that warm, fuzzy feeling.

Obviously a provider has to protect themselves and have a way to drop features/services should there be insufficient demand.  From the above it seems there are two ‘outs’ for Amazon.  The first is contained in Section 3.3 and the other is if they simply terminate the program.  Lets look at Section 3.3

3.3.2. Paid Services (other than Amazon FPS and Amazon DevPay). We may suspend your right and license to use any or all Paid Services (and any associated Amazon Properties) other than Amazon FPS and Amazon DevPay, or terminate this Agreement in its entirety (and, accordingly, cease providing all Services to you), for any reason or for no reason, at our discretion at any time by providing you sixty (60) days’ advance notice in accordance with the notice provisions set forth in Section 15 below.

If you want to use AWS for Disaster Recovery, you better have a plan B as Amazon will only give you 60 days notice if they decide to drop Reserved Instances. And there was I, thinking DR was a plan B…  For some businesses, this may be enough of a commitment, but the more enterprise ’skin’ we put in the Cloud the more we need firmer commitments from providers.  I hope Amazon revises this decision and extends their commitment closer to that of traditional DR providers.

Potential Cost Savings

To give an idea of the cost savings, Geva Perry put together a useful online calculator where you can plug in the compute hours you need and understand the financial savings you’ll see. You can opt for the 1 year or 3 year reservation periods:

Geva goes on to higlight some common scenarios:

The default number I put in the hours column, 8760, is 24×365. So if, for example, you run a Large AMI for the entire year non-stop, you will save $1,153 by using reserved instances.

If you are curious to know the break-even point, it is 4,643 hours annually for the 1-Year fee and 2,381 hours annually for the 3-Year fee. In other words, if you expect to run an instance for more than 4,643 hours during the coming 12 months (works out to an average of 12 hours a day), you’re better off with Reserved, otherwise, stick to On-Demand.

This brings AWS pricing down to that of many traditional hosting providers, for those willing to commit upfront.

You might be thinking “what’s the benefit?” over just paying a traditional non-Cloud provider.  In the next post, I’ll cover some security use cases where lower costs and elasticity come into play.

Oh and if anyone from the AWS team thinks I’ve misinterpreted their Customer Agreement feel free to clarify in the comments and I’ll update the post.

One of the security challenges of Cloud Computing - and specifically Infrastructure as a Service (IaaS) is securely connecting your enterprise network to one or more Cloud providers without deploying VPN hardware.  There is also the availability concern - how quickly can you fail over to another Cloud provider when your primary Cloud evaporates.

One company that I voted ‘Hot’ during my somewhat tongue-in-cheek ‘Hot or Not’ quiz at the World Cloud Computing Summit is trying to do something about it.  CohesiveFT released the first version of VPN-Cubed last year and are now seeking beta testers for their next major release.

What is VPN-Cubed?

VPN-Cubed™ is the first commercial solution that enables customer control in a cloud, across multiple clouds, and between private infrastructure and the clouds.

VPN-Cubed provides an overlay network that allows YOU control of addressing, topology, protocols, and encrypted communications for YOUR devices deployed to virtual infrastructure or cloud computing centers.  When using public clouds your corporate assets are going into 3rd party controlled infrastructure.  This could be public clouds like Amazon EC2.  It could be “gated community” clouds from Telcos like BT, ATT and more.  In both cases you are deploying to 3rd party control, yet Enterprise checks and balances require you to exhibit control over your computing infrastructure.  VPN-Cubed gives you flexibility with control in 3rd party environments.

Despite our product having the word “vpn”  (virtual private network) in its name, VPN-Cubed is more than a simple VPN, it is an overlay network that is configured as easily as a traditional VPN.  X-cloud control, administrative simplicity.  To quote one of the leading security bloggers “this is not your father’s VPN”.

Now you can confidently leverage the cloud for redundancy, failover and scalability during critical transitions; whether scaling up to grow the business or scaling down to cut costs.

If you are seeking to leverage IaaS and want to experiment with the only off-the-shelf Cloud network security overlay, you may want to join the beta test program that’s just opened up by emailing vpncubed_beta@cohesiveft.com.

Update: An overview of the beta program with slides showing screenshots of the steps is now available.

P.S The blogger behind the “this is not your father’s VPN” quote is my friend and fellow Information Security Geek Christofer Hoff over at Rational Survivability.  I reference Chris in my resources page - if you are not already subscribed to his blog, I highly recommend you do.

If you surfed up to the Amazon Web Services homepage today, you’d be forgiven thinking all has been well in the Amazon cloud.

Recent news stories highlight new features and capabilities, including a SQL-like SELECT API for Amazon SimpleDB and the (significant news) that Cloud Compute is now available in the EU.

More worryingly, however and missing from the Amazon AWS homepage, is that Amazon just rolled out an important security fix to correct a basic, but significant, cryptographic weakness in their request signing code.  The flaw affects their database API (SimpleDB) and compute API (EC2) services.

Colin Percival discovered the weakness and reported it to Amazon back in May (!).  Thats 7.5 months ago.  Here’s the “executive summary” from Colins’ write-up:

AWS signature version 1 is insecure

The important bit first: If you are making Query (aka REST) requests to Amazon SimpleDB, to Amazon Elastic Compute Cloud (EC2), or to Amazon Simple Queue Service (SQS) over HTTP, and there is any way for an attacker to provide you with data which you use to construct your request, switch to HTTPS or start using AWS signature version 2 now. For example, if you allow users to add arbitrary “tags” to documents, and you use SimpleDB to store those tags, this means you. (Amazon Flexible Payments Service (FPS) and Amazon Devpay also use the same insecure signature method, but they already require the use of HTTPS. Amazon S3 and other services use different signature methods.)

So in a nutshell, if you only ever accessed SimpleDB or EC2 services over HTTPS, you’re not impacted.

But if you did use the non-encrypted transport, you were vulnerable to a man-in-the-middle attack (MITM) that could expose your data hosted in SimpleDB and your EC2 compute instances.

Overall, Colin is generous in his praise of the response from the Amazon team - although did express concern at the time it took for the vulnerably to be closed:

I reported this issue to Amazon via an email to Jeff Barr, the “Lead Web Services Evangelist” at Amazon on May 1st, and while it took a long time — 7.5 months — for it to be fixed, I’m happy to say that Amazon took this issue seriously at all times, and the lengthy timeline was simply because of the large amount of work involved. Jeff forwarded my email to someone working on SimpleDB (I’ve been asked not to mention names), who confirmed that they agreed that this was a problem. As part of their review of my findings, Amazon’s security people realized that this also affected EC2 and SQS — in my initial investigation I had only looked at SimpleDB — and at the beginning of July they agreed to send me their planned signature version 2 so that I could review it.

Based on the timeline provided by Colin, it took about 2 months for Amazon to assess the extent of the exposure across their other service APIs and come up with a proposed fix.  That’s a considerable length of time given the simplicity of the weakness.  They then took over 5 months to actually implement the fix.  I don’t care how many client libraries you offer Mr Cloud Provider, thats still a very, very long time to leave your customers exposed while you get a fix into production.

Back to Colin as he describes the vulnerable signing process:

AWS signature version 1 signs an HTTP query string as follows:

1. Split the query string based on ‘&’ and ‘=’ characters into a series of key-value pairs.
2. Sort the pairs based on the keys.
3. Append the keys and values together, in order, to construct one big string (key1 + value1 + key2 + value2 + … ).
4. Sign that string using HMAC-SHA1 and your secret access key.

Did you spot the problem?  Colin did and explains as follows:

When Amazon invented this signature scheme, they forgot about one of the foremost design principles relating to cryptographic signatures: Collisions are BAD! In a well-designed signature system, it should be computationally infeasible to construct two different messages which have the same signature; this prevents substitution attacks where an attacker convinces the key holder to sign a “harmless” message, and then attaches that signature to a different message. Looking at how AWS signature version 1 is computed, it’s easy to see how to construct collisions: Because there are no delimiters between the keys and values, the signature for “foo=bar” is identical to the signature for “foob=ar”; moreover, the signature for “foo=bar&fooble=baz” is the same as the signature for “foo=barfooblebaz”.

Oops.  He then goes on to explain how a man in the middle attack could exploit the weakness.

Taking a step back, what are the concerns with this weakness and the way Amazon handles security vulnerability reports in general?

Vulnerability Reporting

• Amazon doesn’t provide a visible mechanism for researchers/customers to report security vulnerabilities.  You can report ‘abuse’ (i.e. your EC2 image is under attack) via a contact form on the AWS website but unless you know someone at Amazon, you have to guess how to report a security vulnerability.
• Amazon has no published policy on how they handle vulnerability reports.  Nothing to explain their commitment to following up, promptly responding to reports, overall process etc.  Now, it seems they do follow up and overall Colin says good things but that timeline sends shivers up my spine.
• Amazon doesn’t have a dedicated security page, mailing list, RSS feed or pigeon carrier for alerting customers of security issues.  This makes it a PITA for enlightended customers to even know they are at  risk (unless they setup  Google News Alerts) and comes across to me at least as a trifle arrogant.

Vulnerability Notification

• Amazon hasn’t notified their customers yet that their SimpleDB and EC2 instances were vulnerable for the past 7.5 months (I’m a customer and I’ve yet to receive an email).  Obviously the vulnerability itself goes back further than 7.5 months.
• Amazon have not published how customers can determine if the issue was exploited (back to the visibility issue again).   Clearly, Amazon doesn’t have full visibliity of the network between the customer and their data centers, however what analysis has Amazon performed on the logs *only they have access to* that could highlight an issue? (frankly, due to the nature of this issue and the lack of event logs/audit trails I suspect the simple answer is there isn’t much they can do here).

Secure Software Development

• Amazons’ internal code review process failed to detect a basic cryptography flaw.  Note, although the error is pretty basic, errors do happen.  The real problem here is the failure of peer review processes to identify the problem prior to going into production.  Cryptography routines along with other sensitive security routines should be subject to greater security analysis as clearly the stakes are higher.  If they did in fact have this code reviewed for security weaknesses, this is a huge oversight.  So this is now fixed, what else is vulnerable?  It would be great to learn that the lengthy delay in delivering the fix was due to Amazon have all their source code reviewed for security issues…
• Amazon have made no public commitment regarding how security is factored into their software development lifecycle.  Until now they could argue they didn’t need to - they hadn’t messed up (or at least, we don’t think they did).

The geek in me loves Amazon AWS and the promise that it holds, but the Security Professional rooted in the “here and now” is seeing too many red flags (the issues are wider than the handling of this one bug).

If anyone from Amazon reads this and wants to talk security roadmap and go “beyond the whitepaper”, you know where I am.