Cloud Security

Regular readers will know I recently extended an invitation to give feedback from an enterprise IT security perspective to any startup that got accepted to present at Under The Radar on April 24th in Mountain View, California.

This post is a summary of a very interesting call I had with George Reese, CTO of enStratus and author of the forthcoming “Cloud Application Architectures” book.  Please note: this isn’t a comprehensive review of the full service, rather it reflects the pieces that we delved into based on some of the common concerns we have around Cloud Security (to give you some idea, we spoke for over 90 minutes…).

enStratus offers cloud infrastructure management tools “aimed at the needs of enterprise IT”.  Today, they support Amazon EC2, with support for other clouds to follow.

Their tag line is ‘Confidence in the Cloud’ and their offering focuses on 3 key areas addressing the twin cloud adoption barriers of security and reliability:

• protecting cloud based data through encryption
• offering service levels above that of the underlying cloud provider (99.9999% for EC2)
• achieving Recovery Time and Recovery Point Objectives “in the face of the most extreme disasters”.

George outlined 3 concerns his customers have about cloud providers such as Amazon:

• Amazon controls the physical systems on which the data resides, meaning Amazon malfeasance, Amazon misfeasance, or even 3rd party subpoenas put that data at risk.
• The complexity of resource orchestration in the context of credential management; i.e. when do your credentials need to be in the cloud versus when their presence is just a security risk
• User management, even via traditional identity management systems, can be dysfunctional.

The enStratus Approach to Cloud Key Management

One of my pet peeves with AWS is the “one key to rule them all” security model (the dysfunctional user management George alluded to).  Any disclosure of that key results in an attacker gaining access to all your infrastructure.  But to make privileged API calls, every developer must have a copy of the key…

Its not unknown for AWS users bundling an AMI (creating a virtual machine image) for public consumption to leave their AWS credentials in the AMI itself.  Oops. This is obviously a Bad Thing ™ as a malicious user that opts to use that AMI can steal their access key, gain access to their Amazon hosted infrastructure and run up bills in their name.

One of the things I really like about the enStratus offering, is the relentless focus on controlling the use and hence exposure of a customers’ ‘cloud masterkeys’. Their implementation keeps the keys away from the AMI, and therefore the cloud, PLUS out of the hands of an org’s IT/dev people.

enStratus acts as a trust broker. After signing up for the service, the customer loads their “all powerful” Amazon credentials via a shared enStratus Provisioning Server into a Credentials Server (no direct internet connectivity).

From that point forward, the customers’ IT people access the enStratus service and manage their cloud infrastructure via named user accounts assigned specific privilege levels.

Permissions include;

• administrator
• start/stop servers
• uptime retrieval and
• audit trail review.

Non-administrative users have no direct access to the AWS keys.

Here’s a peek at the architecture of enStratus.

When an authorised enStratus user issues cloud infrastructure management requests via the Web Services and Console server, the Provisioning server issues the cloud API calls on behalf of the users.  This eliminates the need for every user needing a copy of the key to sign requests.  Given they are mediating API requests, adding logging functionality was a no-brainer and means the next time you need to know ‘who spun up that unpatched AMI image with an allow-all security group?’, you can find out.

Its important to note that there is nothing preventing anyone with your AWS key from just making API calls directly to the AWS API endpoint - totally bypassing the enStratus infrastructure.  Therefore, careful key lifecycle management is still necessary; i.e. load fresh AWS credentials straight into enStratus and follow a “no sharing” policy.

I should point out that the EC2 ecosystem players can’t do anything about this as the AWS platform doesn’t offer a mechanism to tie IP level controls to AWS key usage or EC2 (yet).  One way Amazon could implement this (nothing announced) is with their new JSON based Access Policy Language.  Despite this, enStratus can still detect new EC2 instances spun-up by API calls they didn’t mediate, through telemetry used for operational monitoring - they just won’t be able to tell you who started it.

enStratus can help customers build their AMIs, including bundling in HIDS (Host based Intrusion Detection) via ossec, with centralised agent reporting.  Another example of how they protect the AMI key is through error checking in their scripted AMI builds to ensure key material is not left in an AMI accidently.  In addition, users are prevented from accessing partially provisioned AMIs (to eliminate potential key snarfing shenanigans).

Root access to EC2 images is disabled by default (unlike with vanilla EC2).  Privileged access is granted via sudo.

Filesystem Encryption

enStratus offers optional filesystem encryption through a checkbox.  Keys are temporarily passed into the EC2 instance when required; i.e. mounting.

Encrypted filesystem support is implemented via 2 block volumes configured as RAID 0.  2 sets of encryption keys are used.  One for encrypting and remounting the ephemeral drive (this is a “non-persistent store” automagically attached by EC2 to each running AMI).  The second key pair is used to encrypt and mount filesystems attached as Elastic Block Storage (EBS).  EBS is off-instance, persistent storage.  To avoid potential exposure of key material, the 2nd set of keys are stored on the encrypted ephemeral drive during mount.

Worth noting is that in testing, George found that 2 EBS volumes, configured as RAID 0 with an encrypted XFS filesystem  offers similar performance to a single, unencrypted EBS volume with an ext3 filesystem.

George is keen to stress that enStratus is not looking to control both customers data and their keys.  So whilst he recommends and can help customers make use of the EBS snapshot feature to clone/backup storage volumes to Amazon S3 (Simple Storage Service), he isn’t offering a hosted backup service (to avoid a potential conflict).  Of course, an evil and privileged enStratus employee could access your live data as the keys are stored in their Credential server.  Today though, enStratus is a small company so figuring out ‘who dunnit’ would not require the services of Sherlock Holmes.

Futures

Today, the enStratus management infrastructure sits outside of the cloud (at a colo) for cloud monitoring and isolation reasons.  George is exploring the possibility of also offering an on-premise offering for customers wishing for more control.

I offered a few short and medium term suggestions around additional integrity checks, encryption ideas, assurance processes (source code security reviews, penetration testing) and consideration to using a Hardware Security Module (HSM) for key storage to further bolster both security and trust.  George seemed genuinely open and receptive to these ideas and also shared a few interesting customer requests they are actively working on today.  Some of the more expensive line items would become practical if they can attract additional funding.

Overall, I have to say I’m impressed with their approach, technology and attitude.  Definitely worth a hands-on evaluation if current Cloud controls don’t fall within your risk tolerance.

Good luck to George and the rest of the enStratus team as they prepare to present at Under The Radar!

Attending Under The Radar?

As a special offer to cloudsecurity.org readers, the organisors of Under The Radar are offering $100 off the list price for VIP tickets.  To claim yours, click here.

Positive news from the Amazon camp today as Jeff Barr from the AWS team announces a new access control policy.  Right now, its applicable to the Simple Queue Service (SQS).

A Quick SQS Reminder

For those unfamiliar with SQS, here’s the elevator pitch from Amazon:

Amazon Simple Queue Service (Amazon SQS) offers a reliable, highly scalable, hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed components of their applications that perform different tasks, without losing messages or requiring each component to be always available. Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud EC2 and the other AWS infrastructure web services.

Amazon SQS works by exposing Amazon’s web-scale messaging infrastructure as a web service. Any computer on the Internet can add or read messages without any installed software or special firewall configurations. Components of applications using Amazon SQS can run independently, and do not need to be on the same network, developed with the same technologies, or running at the same time.

So, a very handy data structure that makes perfect sense in distributed programming.  However, access control options were limited…until today.

The New Secret Sauce

AWS is also introducing additional permission features that control access to Amazon SQS and to each of its fundamental actions on a very fine-grained basis. You can exercise this control at two levels:

* At the higher level you can use the new AddPermission and RemovePermission functions to set and remove particular access rights for each queue. Access rights, including the ability to send, receive, or delete messages, change message visibility, or to retrieve queue attributes, can be granted to any AWS user via their AWS account number.
* At the lower level you can use our new Access Policy Language. This expressive language makes its debut as part of this SQS release; over time, we plan to employ this Access Policy Language with our other services. The Access Policy Language enables the creation of complex rules to enable access to queues based on identity (AWS account number), source IP address, date, time, and more.

With this new permission system you can now use Amazon SQS queues to connect non-AWS applications to AWS applications and to connect AWS applications from different organizations. You could use an open, public queue as a drop box, allowing outside applications to submit work items for processing. This could be a fully public drop box, or it could be limited to requests from a single country by using a policy based on an IP address or address range. Communication between organizations can be established based on IP addresses or AWS accounts, as appropriate.

For me, the most significant news is not so much that SQS now has fine grained access control, but that Amazon have introduced a Access Policy Language and they plan to apply it to other AWS services.  This is a very positive development and could be the mechanism they use to overcome some of the longstanding security concerns I blogged about recently.

Architectural Overview

For the visually inclined (source):

Where:

1. You, the resource owner.

2. Your resources (contained within the AWS service; e.g., SQS queues).

3. Your policies.  Typically you have one policy per resource, although you could have multiple. The AWS service itself provides an API you use to upload and manage your policies. For information about the content of the policies, see How to Write a Policy.

4. Requesters and their incoming requests to the AWS service.

5. The access policy language evaluation code.

This is the set of code within the AWS service that evaluates incoming requests against the applicable policies and determines whether the requester is allowed access to the resource. For information about how the service makes the decision, see Evaluation Logic (Ed: note there are soft and hard denials).

An Example

Here’s an example from the developer docs showing a simple IP based control (multiple controls can be applied to a single resource):

The following example policy gives all users permission to use all possible SQS actions that can be shared for the queue named 987654321098/queue1, but only if the request comes from the 192.168.143.0/24 range.

{
  "Version": "2008-10-17",
  "Id": "Queue1_Policy_UUID",
  "Statement":
     {
        "Sid":"Queue1_AnonymousAccess_AllActions_WhitelistIP",
        "Effect": "Allow",
        "Principal": {
           "AWS": "*"
         },
        "Action": "SQS:*",
        "Resource": "/987654321098/queue1",
        "Condition" : {
            "IpAddress" : {
               "SourceIP":"192.168.143.0/24"
            }
     }
}

Conclusion

Notice the values for the ‘Action’ and ‘Resource’ tags.  Now imagine those with different AWS service identifiers and resource types and things start to get really interesting.

Now all we need is an user-friendly, hard-to-shoot-yourself-in-the-foot policy generator/front-end to open this feature up to the masses.

All in all, its great to see the introduction of a consistent policy language from the cloud pioneer.

I’m off to learn more about the language…

Update: in case it isn’t obvious from the example, the policy language is expressed using JSON (thanks @lmacvittie for the prompt)

Yesterday, Amazon released a whitepaper called ‘Creating HIPAA-compliant Medical Data Applications with AWS’.  There is plenty to comment on, but for now I want to draw attention to some security concerns I have around AWS and thus its current readiness to handle data that falls under HIPAA.

I’m a big fan of Amazon AWS - they are true innovators and the diversity of their service offerings is astonishing.  There are many use cases for Cloud Computing and not all have high assurance needs. But when touting HIPAA compliance for the processing and storage of medical data, I can’t help feeling there are (amongst other things) glaring API level control gaps that would make me extremely uncomfortable with my medical data stored in the Amazon Cloud today.

Amazon is marketing their Cloud - with its current security controls - as able to support HIPAA compliant applications.  The whitepaper does a good job talking up their existing security controls but omits to mention a number of serious control gaps that decision makers should be aware of prior to committing “other peoples data” medical data to the Amazon Cloud.

I’m not a HIPAA expert and therefore in my ignorance, it may well be that AWS can satisfy covered entities security requirements (if so, it says more about HIPAA than AWS or, more about a specific use case).  Consequently, I’ll step aside from the ‘compliance’ angle for a moment and confine my comments to ’security’ and encourage those with first hand HIPAA compliance experience to comment as they see fit.

There is no customer accessible AWS API call audit log

In other words, you have no way to know if, when and from where (source IP) your AWS key was used to make API calls that may affect the security posture of your AWS resources (an exception is S3, but only if you turn on logging (off by default)). For example; if your key is compromised without your awareness and your EC2 “security groups” (firewall) are changed, there is no customer accessible log record. The only way to detect this is to poll the API on a regular basis (DescribeSecurityGroups) and compare with the output of previous calls or to call up Amazon support on a regular basis and ask them “is all well?”  (j/k).  I can think of much more sinister API calls to invoke, the security groups is just an example.

They also afford no customer visibility over failed authentication attempts at the AWS web interface or at the API.  Again, no visibilty to understand if your infrastructure in the sky is under attack.  Fine for some workloads, not so fine for others.

Oh, and while we are at it, why is there no firewall log for the built in ’security groups’ firewall functionality?  To get visibility today, you have to run a host based firewall with logging enabled but even then you will not see what was dropped by Amazon (when doing network security investigations, this information can be really helpful).  Basic reporting accessible to customers would be a good start.

There is no way to restrict the source IP address from which the sacred AWS API key can be used from

The AWS API interface can be used from any source IP at any time (and as above, you have no audit trail for EC2 API calls).  This is equivalent of exposing your compute and storage management API to the entire planet.  To be clear, they are not exposing their *internal* management plane - the infrastructure that the Amazon AWS team uses to manage the AWS infrastructure, rather its *your* hosted infrastruture management plane that is exposed.

Why can’t the EC2 security group concept (aka firewall) be applied to the AWS API, to limit access from approved management stations? After creating my key I should have the option to limit use to specific subnets of my organisation.  I’d love to hear this option is on the Amazon security roadmap (along with visibility of failed attempts).

Each AWS account is limited to a single key - exposure results in total security failure

I’ve said it before and I’ll say it again - you only get a single key per AWS account.

That makes it really hard to segregate environments.  To do so, you have to create multiple AWS accounts with different credit cards and even then that only gives a very coarse grained access model.

With your master key, why can’t you generate a handful of subkeys for different use cases?

If you could link those subkeys to roles that reflect your security policies, then you’d really have flexibility.  This would also mean you could lock away the all-powerful master key for use only when such power is warranted (thus reducing potential exposure).

The failure mode of a single key is ‘complete and utter’.  If my key is compromised, my entire AWS environment is compromised - all my virtual machines, all my data.  I don’t know about you, but that scares the living daylights out of me in the context of medical data.

Takeaway

Hopefully we all know by now that “compliance” does not equal “security”, but HIPAA compliance not withstanding, would you really want your medical data in a Cloud without some or all of these fundamental control gaps resolved or mitigated?  I can’t find anything in this new whitepaper or the old AWS Security Whitepaper that speaks directly to these issues.  If I missed something, please share below and I’ll update the post.

When medical data is stored at the medical provider, I can believe they will have a firewall that doesn’t allow outsiders to make API calls to their compute and storage management plane from anywhere in the world…  I can believe that leakage of an internal infrastructure management password cannot be used from *outside* the firewall.  At least, not without first exploiting an unrelated security weakness to gain access to the environment in the first place.

I can appreciate that providers are keen to demonstrate use cases but I can’t help feeling that in their eagerness to market ‘compliance’, Amazon is putting the cart before the horse from a security perspective.

What do you think?

Put simply, these are virtual machine instances that you can pay to have reserved for you to use anytime you want. The benefit should be assured availability and lower cost for machines with heavy uptime requirements.  You still only pay for what you use, in addition to a one time non-refundable payment to make the reservation.

When I first read about this I got quite excited.  I was already thinking of the potential use cases from a security perspective, particularly around business continuance/disaster recovery.

In fact, this is something fellow AWS customers have been asking for:

Also, quite a few customers actually told us something even more interesting: they were interested in using EC2 but needed to make sure that we would have a substantial number of instances available to them at any time in order for them to use EC2 in a DR (Disaster Recovery) scenario. In a scenario like this, you can’t simply hope that your facility has sufficient capacity to accommodate your spot needs; you need to secure a firm resource commitment ahead of time.

This is where it gets interesting.  What does ‘firm resource commitment ahead of time’ mean?

Always Read The Label

I surfed over to the AWS Customer Agreement to read the small print:

8.3. Reserved Instance Pricing. You may designate EC2 instances as subject to the reserved pricing and payment terms (“Reserved Instance Pricing”) set forth on the EC2 detail page on the AWS Website (each designated instance, a “Reserved Instance”). You may designate instances as Reserved Instances solely by calling to the Purchasing API (the “API Call”). In the API Call you must designate an availability zone, instance type and quantity for the applicable Reserved Instances. The Reserved Instances may only be used in the designated availability zone. We may change Reserved Instance Pricing at any time but price changes will not apply to previously designated Reserved Instances. We may terminate the Reserved Instance Pricing program at any time. Notwithstanding anything to the contrary herein, Reserved Instances are nontransferable and all amounts paid in connection with the Reserved Instances are nonrefundable, except that if we terminate this Agreement pursuant to Section 3.3 or terminate the Reserved Instance Pricing program we will refund you a pro rata portion of any up-front fee paid in connection with any previously designated Reserved Instances. In addition to being subject to Reserved Instance Pricing, Reserved Instances are subject to all data transfer and other fees applicable under this Agreement.

Hmm, suddenly I lost that warm, fuzzy feeling.

Obviously a provider has to protect themselves and have a way to drop features/services should there be insufficient demand.  From the above it seems there are two ‘outs’ for Amazon.  The first is contained in Section 3.3 and the other is if they simply terminate the program.  Lets look at Section 3.3

3.3.2. Paid Services (other than Amazon FPS and Amazon DevPay). We may suspend your right and license to use any or all Paid Services (and any associated Amazon Properties) other than Amazon FPS and Amazon DevPay, or terminate this Agreement in its entirety (and, accordingly, cease providing all Services to you), for any reason or for no reason, at our discretion at any time by providing you sixty (60) days’ advance notice in accordance with the notice provisions set forth in Section 15 below.

If you want to use AWS for Disaster Recovery, you better have a plan B as Amazon will only give you 60 days notice if they decide to drop Reserved Instances. And there was I, thinking DR was a plan B…  For some businesses, this may be enough of a commitment, but the more enterprise ’skin’ we put in the Cloud the more we need firmer commitments from providers.  I hope Amazon revises this decision and extends their commitment closer to that of traditional DR providers.

Potential Cost Savings

To give an idea of the cost savings, Geva Perry put together a useful online calculator where you can plug in the compute hours you need and understand the financial savings you’ll see. You can opt for the 1 year or 3 year reservation periods:

Geva goes on to higlight some common scenarios:

The default number I put in the hours column, 8760, is 24×365. So if, for example, you run a Large AMI for the entire year non-stop, you will save $1,153 by using reserved instances.

If you are curious to know the break-even point, it is 4,643 hours annually for the 1-Year fee and 2,381 hours annually for the 3-Year fee. In other words, if you expect to run an instance for more than 4,643 hours during the coming 12 months (works out to an average of 12 hours a day), you’re better off with Reserved, otherwise, stick to On-Demand.

This brings AWS pricing down to that of many traditional hosting providers, for those willing to commit upfront.

You might be thinking “what’s the benefit?” over just paying a traditional non-Cloud provider.  In the next post, I’ll cover some security use cases where lower costs and elasticity come into play.

Oh and if anyone from the AWS team thinks I’ve misinterpreted their Customer Agreement feel free to clarify in the comments and I’ll update the post.

On the 23rd June, Amazon quietly rolled out a security fix for an issue originally discussed in the Amazon developer forums. Amazon documentation was revised to reflect the change as follows:

“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”

Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).

What does this mean?

It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only if you manually verify the host SSH key during your initial SSH connection.

OK, but I created my AMI before June 23rd – am I vulnerable?

According to Amazon, yes.  Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI.  The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.

How can I fix my pre-June 23rd AMIs?

Regenerate the SSH host key.  The exact commands will depend on your operating system (hint: ssh-keygen).

Who is to blame?

Either the creators of the original AMI or Amazon – depends how you look at it.  If Amazon created the public AMI then it could be argued they are responsible.  However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).

Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image).   That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?

Just to be clear, its not the fault of SSH – ’secure channels’ require proper key management and the need for unique host keys is well documented.

Are there any mitigating factors?

Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet).  You’ll still want to regenerate the ssh host keys sooner than later.

Is the Amazon environment vulnerable to Man-in-the-middle attacks?

I don’t know.  But that isn’t the real question – is the path between you and your AMI immune to MITM attacks and the answer is most definitely no.  If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-).  From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).

Why Didn’t Amazon Tell Me I’m Vulnerable?  They know from their logs what AMIs I use!

Didn’t they?  Whoops – naughty Amazon :P.

But seriously, Amazon are not responsible for the configuration of the public AMIs you use.  Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.

Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).

Not that I know of.   Right now you have to search forum posts and monitor documentation updates – which is time consuming and makes it easy to miss something.  I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management).   In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security  (for some customers, if they don’t “see it”, it doesn’t “exist”).

A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.