Cloud Security

Amazon just announced "Spot Pricing" for their EC2 instances:

"Spot Instances are a new way to purchase and consume Amazon EC2 Instances. They allow customers to bid on unused Amazon EC2 capacity and run those instances for as long as their bid exceeds the current Spot Price. The Spot Price changes periodically based on supply and demand, and customers whose bids meet or exceed it gain access to the available Spot Instances. Spot Instances are complementary to On-Demand Instances and Reserved Instances, providing another option for obtaining compute capacity.

For customers with flexibility in when their applications can run, Spot Instances can significantly lower their Amazon EC2 costs. Additionally, Spot Instances can provide access to large amounts of additional capacity for applications with urgent needs. Just a few examples of categories of applications well-suited to Spot Instances are:

* Image and video processing, conversion and rendering
* Scientific research data processing
* Financial modeling and analysis"

This is innovative and in a world of "fair play" is a smart move by Amazon. But lets consider this from a Black Hat hacker perspective for a moment:

Black Hats have all kinds of reasons for needing massive compute power. The obvious one is password cracking and that can take an awful lot of CPU power...

But bad guys feel the credit crunch too and don't like to pay list price even when its not their money (think stolen credit cards). Spot prices for EC2 instances are discounts on regular EC2 pricing for all. The downside is, you have to wait until the spot price is equal to, or lower than your maximum bid. Some people won't wait for "the market" - they want their goodies faster. Since the market is nothing more than current utilization level of EC2, the spot price goes down when EC2 is less utilized. As a Black Hat, you might want to influence that to drive the spot price down. Suddenly all those DoS attacks that you could only use to either cripple your online enemies or bribe gambling sites and the like, have a new use case - crashing existing customers EC2 instances to artificially decrease demand for EC2 instances and thus lower the spot price.

Ouch.

The counterargument is that Black Hats won't bother going to all this effort - they don't need to. They already have access to enough disposable credit lines and/or vulnerable machines that they just won't bother. It's hard to disagree today. In Amazon PR terms, this is more a 'theoretical' attack - for now.

Regardless, the downside is that the introduction of an innovative pricing model today by Amazon, leaves existing EC2 customers more exposed to DoS attack than they were before.

Update:

Amazon note in their updated AWS Customer Agreement:

"You may not, directly, indirectly, alone or in cooperation with any third party, attempt to control, influence or manipulate the price for Spot Instances. Without limiting the foregoing, you may not submit requests for Spot Instances through any third party (e.g., “proxy bidding”) or share information with any third party regarding the maximum prices specified in your Spot Instance Requests".

Last week, saw the release of a research paper from the University of California and MIT called Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds [PDF].

The abstract reads:

Third-party cloud computing represents the promise of outsourcing as applied to computation. Services, such as Microsoft’s Azure and Amazon’s EC2, allow users to instantiate virtual machines (VMs) on demand and thus purchase precisely the capacity they require when they require it.  In turn, the use of virtualization allows third-party cloud providers to maximize the utilization of their sunk capital costs by multiplexing many customer VMs across a shared physical infrastructure. However, in this paper, we show that this approach can also introduce new vulnerabilities.
Using the Amazon EC2 service as a case study, we show that it is possible to map the internal cloud infrastructure, identify where a particular target VM is likely to reside, and then instantiate new VMs until one is placed co-resident with the target. We explore how such placement can then be used to mount cross-VM side-channel attacks to extract information from a target VM on the same machine.

After introducing the main concepts the threat model of the research is defined:

In our threat model, adversaries are non-provider-affliated malicious parties. Victims are users running confidentiality-requiring services in the cloud. A traditional threat in such a setting is direct compromise, where an attacker attempts remote exploitation of vulnerabilities in the software running on the system. Of course, this threat exists for cloud applications as well. These kinds of attacks (while important) are a known threat and the risks they present are understood.

We instead focus on where third-party cloud computing gives attackers novel abilities; implicitly expanding the attack surface of the victim. We assume that, like any customer, a malicious party can run and control many instances in the cloud, simply by contracting for them. Further, since the economies offered by third-party compute clouds derive from multiplexing physical infrastructure, we assume (and later validate) that an attacker’s instances might even run on the same physical hardware as potential victims. From this vantage, an attacker might manipulate shared physical resources (e.g., CPU caches, branch target buffers, network queues, etc.) to learn otherwise confidential information.  In this setting, we consider two kinds of attackers: those who cast a wide net and are interested in being able to attack some known hosted service and those focused on attacking a particular victim service. The latter’s task is more expensive and time-consuming than the former’s, *but both rely on the same fundamental attack.

Section (5) “Network Probing” lays the foundation for (6) “Cloud Cartography” through describing…

…an empirical measurement study focused on understanding VM placement in the EC2 system and achieving co-resident placement for an adversary. To do this, we make use of network probing both to identify public services hosted on EC2 and to provide evidence of co-residence (that two instances share the same physical server)

They then go on to describe the tools and techniques in:

• “enumerating public EC2-based web servers using external probes” (hping2, nmap, wget)
• “translating responsive public IPs to internal IPs (via DNS queries within the cloud)” (DNS)
• “launching a number of EC2 instances of varying types and surveying the resulting IP address assigned” (I assume this was scripted)

Using data from booting 4,499 (!) EC2 instances across different EC2 accounts under their control (remember that by default, an EC2 account has a 20-instance soft-limit), they develop the following heuristics for mapping EC2 instances:

• All IPs from a /16 are from the same EC2 availability zone (e.g. US).
• A /24 inherits any included sampled instance type (e.g. small, large, x-large etc). If there are multiple instances with distinct types, then we label the /24 with each distinct type (i.e., it is ambiguous).
• A /24 containing a DomO IP address only contains Dom0 IP addresses. We associate to this /24 the type of the DomO’s associated instance (note: DomO is the first domain started by the hypervisor after boot)
• All /24’s between two consecutive Dom0 /24’s inherit the former’s associated type.

They conclude this section with “cartography countermeasures”.  This boils down to making local IP address assignment (from the address pool) random across instance types and availability zones and/or restricting the customers view.

Section (6) describes how to determine co-residence:

Given a set of targets, the EC2 map from the previous section educates choice of instance launch parameters for attempting to achieve placement on the same physical ma- chine. Recall that we refer to instances that are running on the same physical machine as being co-resident.

In this section we describe several easy-to-implement co-residence checks.

Looking ahead, our eventual check of choice will be to compare instances’ Dom0 IP addresses.

We confirm the accuracy of this (and other) co-residence checks by exploit- ing a hard-disk-based covert channel between EC2 instances.

They identify 3 network-based co-residence checks:

• matching Dom0 IP address
• small packet round-trip times
• numerically close internal IP addresses (e.g. within 7).

Now able to determine that a given EC2 instance under their control is on the same physical hardware as the target, section (7) analyses EC2’s VM placement strategy and explores techniques the attacker can use to achieve reliable co-residence with a target VM.   In the context of a brute force placement attack they note that “even a very naive attack strategy can successfully achieve co-residence against a not-so-small fraction of targets”.  Thus if your target set is wide, brute force turns out to be a viable strategy.  For individual or small sets of targets, a more efficient strategy is “instance flooding” (spinning up numerous VMs) immediately after the target has booted to “take advantage of the parallel placement locality exhibited by the EC2 placement algorithms”.  This is where the dynamic nature of cloud comes into play:

But why would we expect that an attacker can launch instances soon after a particular target victim is launched?  Here the dynamic nature of cloud computing plays well into the hands of creative adversaries. Recall that one of the main features of cloud computing is to only run servers when needed. This suggests that servers are often run on instances, terminated when not needed, and later run again.  So for example, an attacker can monitor a server’s state (e.g., via network probing), wait until the instance disappears, and then if it reappears as a new instance, engage in instance flooding. Even more interestingly, an attacker might be able to actively trigger new victim instances due to the use of auto scaling systems. These automatically grow the number of instances used by a service to meet increases in demand. (Examples include scalr ³⁰ and RightGrid ²⁸.  See also ⁶.) We believe clever adversaries can find many other practical realizations of this attack scenario.

With co-residence achieved, section (8) assesses the practicality of side channel attacks in a VM environment.  As you’d expect, a number of possibilities exist, however the holy grail of cryptographic key extraction does not appear plausible (at this time).  One notable quote:

The side channel attacks we report on in the rest of this section are more coarse-grained than those required to extract cryptographic keys. While this means the attacks extract less bits of information, it also means they are more robust and potentially simpler to implement in noisy environments such as EC2.

Side channel attacks discussed include:

• Denial of Service
• Measure cache usage (measure CPU utilisation on the physical machine; or “how busy are their servers?”)
• Load-based co-residence detection (aka detecting co-residence without relying on sending any network probes)
• Estimating traffic rates (sounds harmless but can be used to deduce targets activity patterns, peak trading times for maximal DoS effect etc)
• Keystroke timing attack (remote keystroke monitoring)

As with each of the other sections, the authors suggest potential countermeasures.

Overall the paper makes a very interested read.  There’s no EC2 “0-day”, but that’s not the intent of the paper.  Rather, we are reminded that cloud platforms and technologies do bring some novel attacks that thus far have not really figured in much of the security conversation to date.  We need more of this type of research to better understand what we are getting ourselves into.

Regular readers will know I recently extended an invitation to give feedback from an enterprise IT security perspective to any startup that got accepted to present at Under The Radar on April 24th in Mountain View, California.

This post is a summary of a very interesting call I had with George Reese, CTO of enStratus and author of the forthcoming “Cloud Application Architectures” book.  Please note: this isn’t a comprehensive review of the full service, rather it reflects the pieces that we delved into based on some of the common concerns we have around Cloud Security (to give you some idea, we spoke for over 90 minutes…).

enStratus offers cloud infrastructure management tools “aimed at the needs of enterprise IT”.  Today, they support Amazon EC2, with support for other clouds to follow.

Their tag line is ‘Confidence in the Cloud’ and their offering focuses on 3 key areas addressing the twin cloud adoption barriers of security and reliability:

• protecting cloud based data through encryption
• offering service levels above that of the underlying cloud provider (99.9999% for EC2)
• achieving Recovery Time and Recovery Point Objectives “in the face of the most extreme disasters”.

George outlined 3 concerns his customers have about cloud providers such as Amazon:

• Amazon controls the physical systems on which the data resides, meaning Amazon malfeasance, Amazon misfeasance, or even 3rd party subpoenas put that data at risk.
• The complexity of resource orchestration in the context of credential management; i.e. when do your credentials need to be in the cloud versus when their presence is just a security risk
• User management, even via traditional identity management systems, can be dysfunctional.

The enStratus Approach to Cloud Key Management

One of my pet peeves with AWS is the “one key to rule them all” security model (the dysfunctional user management George alluded to).  Any disclosure of that key results in an attacker gaining access to all your infrastructure.  But to make privileged API calls, every developer must have a copy of the key…

Its not unknown for AWS users bundling an AMI (creating a virtual machine image) for public consumption to leave their AWS credentials in the AMI itself.  Oops. This is obviously a Bad Thing ™ as a malicious user that opts to use that AMI can steal their access key, gain access to their Amazon hosted infrastructure and run up bills in their name.

One of the things I really like about the enStratus offering, is the relentless focus on controlling the use and hence exposure of a customers’ ‘cloud masterkeys’. Their implementation keeps the keys away from the AMI, and therefore the cloud, PLUS out of the hands of an org’s IT/dev people.

enStratus acts as a trust broker. After signing up for the service, the customer loads their “all powerful” Amazon credentials via a shared enStratus Provisioning Server into a Credentials Server (no direct internet connectivity).

From that point forward, the customers’ IT people access the enStratus service and manage their cloud infrastructure via named user accounts assigned specific privilege levels.

Permissions include;

• administrator
• start/stop servers
• uptime retrieval and
• audit trail review.

Non-administrative users have no direct access to the AWS keys.

Here’s a peek at the architecture of enStratus.

When an authorised enStratus user issues cloud infrastructure management requests via the Web Services and Console server, the Provisioning server issues the cloud API calls on behalf of the users.  This eliminates the need for every user needing a copy of the key to sign requests.  Given they are mediating API requests, adding logging functionality was a no-brainer and means the next time you need to know ‘who spun up that unpatched AMI image with an allow-all security group?’, you can find out.

Its important to note that there is nothing preventing anyone with your AWS key from just making API calls directly to the AWS API endpoint - totally bypassing the enStratus infrastructure.  Therefore, careful key lifecycle management is still necessary; i.e. load fresh AWS credentials straight into enStratus and follow a “no sharing” policy.

I should point out that the EC2 ecosystem players can’t do anything about this as the AWS platform doesn’t offer a mechanism to tie IP level controls to AWS key usage or EC2 (yet).  One way Amazon could implement this (nothing announced) is with their new JSON based Access Policy Language.  Despite this, enStratus can still detect new EC2 instances spun-up by API calls they didn’t mediate, through telemetry used for operational monitoring - they just won’t be able to tell you who started it.

enStratus can help customers build their AMIs, including bundling in HIDS (Host based Intrusion Detection) via ossec, with centralised agent reporting.  Another example of how they protect the AMI key is through error checking in their scripted AMI builds to ensure key material is not left in an AMI accidently.  In addition, users are prevented from accessing partially provisioned AMIs (to eliminate potential key snarfing shenanigans).

Root access to EC2 images is disabled by default (unlike with vanilla EC2).  Privileged access is granted via sudo.

Filesystem Encryption

enStratus offers optional filesystem encryption through a checkbox.  Keys are temporarily passed into the EC2 instance when required; i.e. mounting.

Encrypted filesystem support is implemented via 2 block volumes configured as RAID 0.  2 sets of encryption keys are used.  One for encrypting and remounting the ephemeral drive (this is a “non-persistent store” automagically attached by EC2 to each running AMI).  The second key pair is used to encrypt and mount filesystems attached as Elastic Block Storage (EBS).  EBS is off-instance, persistent storage.  To avoid potential exposure of key material, the 2nd set of keys are stored on the encrypted ephemeral drive during mount.

Worth noting is that in testing, George found that 2 EBS volumes, configured as RAID 0 with an encrypted XFS filesystem  offers similar performance to a single, unencrypted EBS volume with an ext3 filesystem.

George is keen to stress that enStratus is not looking to control both customers data and their keys.  So whilst he recommends and can help customers make use of the EBS snapshot feature to clone/backup storage volumes to Amazon S3 (Simple Storage Service), he isn’t offering a hosted backup service (to avoid a potential conflict).  Of course, an evil and privileged enStratus employee could access your live data as the keys are stored in their Credential server.  Today though, enStratus is a small company so figuring out ‘who dunnit’ would not require the services of Sherlock Holmes.

Futures

Today, the enStratus management infrastructure sits outside of the cloud (at a colo) for cloud monitoring and isolation reasons.  George is exploring the possibility of also offering an on-premise offering for customers wishing for more control.

I offered a few short and medium term suggestions around additional integrity checks, encryption ideas, assurance processes (source code security reviews, penetration testing) and consideration to using a Hardware Security Module (HSM) for key storage to further bolster both security and trust.  George seemed genuinely open and receptive to these ideas and also shared a few interesting customer requests they are actively working on today.  Some of the more expensive line items would become practical if they can attract additional funding.

Overall, I have to say I’m impressed with their approach, technology and attitude.  Definitely worth a hands-on evaluation if current Cloud controls don’t fall within your risk tolerance.

Good luck to George and the rest of the enStratus team as they prepare to present at Under The Radar!

Attending Under The Radar?

As a special offer to cloudsecurity.org readers, the organisors of Under The Radar are offering $100 off the list price for VIP tickets.  To claim yours, click here.