Cloud Security

Last week, saw the release of a research paper from the University of California and MIT called Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds [PDF].

The abstract reads:

Third-party cloud computing represents the promise of outsourcing as applied to computation. Services, such as Microsoft’s Azure and Amazon’s EC2, allow users to instantiate virtual machines (VMs) on demand and thus purchase precisely the capacity they require when they require it.  In turn, the use of virtualization allows third-party cloud providers to maximize the utilization of their sunk capital costs by multiplexing many customer VMs across a shared physical infrastructure. However, in this paper, we show that this approach can also introduce new vulnerabilities.
Using the Amazon EC2 service as a case study, we show that it is possible to map the internal cloud infrastructure, identify where a particular target VM is likely to reside, and then instantiate new VMs until one is placed co-resident with the target. We explore how such placement can then be used to mount cross-VM side-channel attacks to extract information from a target VM on the same machine.

After introducing the main concepts the threat model of the research is defined:

In our threat model, adversaries are non-provider-affliated malicious parties. Victims are users running confidentiality-requiring services in the cloud. A traditional threat in such a setting is direct compromise, where an attacker attempts remote exploitation of vulnerabilities in the software running on the system. Of course, this threat exists for cloud applications as well. These kinds of attacks (while important) are a known threat and the risks they present are understood.

We instead focus on where third-party cloud computing gives attackers novel abilities; implicitly expanding the attack surface of the victim. We assume that, like any customer, a malicious party can run and control many instances in the cloud, simply by contracting for them. Further, since the economies offered by third-party compute clouds derive from multiplexing physical infrastructure, we assume (and later validate) that an attacker’s instances might even run on the same physical hardware as potential victims. From this vantage, an attacker might manipulate shared physical resources (e.g., CPU caches, branch target buffers, network queues, etc.) to learn otherwise confidential information.  In this setting, we consider two kinds of attackers: those who cast a wide net and are interested in being able to attack some known hosted service and those focused on attacking a particular victim service. The latter’s task is more expensive and time-consuming than the former’s, *but both rely on the same fundamental attack.

Section (5) “Network Probing” lays the foundation for (6) “Cloud Cartography” through describing…

…an empirical measurement study focused on understanding VM placement in the EC2 system and achieving co-resident placement for an adversary. To do this, we make use of network probing both to identify public services hosted on EC2 and to provide evidence of co-residence (that two instances share the same physical server)

They then go on to describe the tools and techniques in:

• “enumerating public EC2-based web servers using external probes” (hping2, nmap, wget)
• “translating responsive public IPs to internal IPs (via DNS queries within the cloud)” (DNS)
• “launching a number of EC2 instances of varying types and surveying the resulting IP address assigned” (I assume this was scripted)

Using data from booting 4,499 (!) EC2 instances across different EC2 accounts under their control (remember that by default, an EC2 account has a 20-instance soft-limit), they develop the following heuristics for mapping EC2 instances:

• All IPs from a /16 are from the same EC2 availability zone (e.g. US).
• A /24 inherits any included sampled instance type (e.g. small, large, x-large etc). If there are multiple instances with distinct types, then we label the /24 with each distinct type (i.e., it is ambiguous).
• A /24 containing a DomO IP address only contains Dom0 IP addresses. We associate to this /24 the type of the DomO’s associated instance (note: DomO is the first domain started by the hypervisor after boot)
• All /24’s between two consecutive Dom0 /24’s inherit the former’s associated type.

They conclude this section with “cartography countermeasures”.  This boils down to making local IP address assignment (from the address pool) random across instance types and availability zones and/or restricting the customers view.

Section (6) describes how to determine co-residence:

Given a set of targets, the EC2 map from the previous section educates choice of instance launch parameters for attempting to achieve placement on the same physical ma- chine. Recall that we refer to instances that are running on the same physical machine as being co-resident.

In this section we describe several easy-to-implement co-residence checks.

Looking ahead, our eventual check of choice will be to compare instances’ Dom0 IP addresses.

We confirm the accuracy of this (and other) co-residence checks by exploit- ing a hard-disk-based covert channel between EC2 instances.

They identify 3 network-based co-residence checks:

• matching Dom0 IP address
• small packet round-trip times
• numerically close internal IP addresses (e.g. within 7).

Now able to determine that a given EC2 instance under their control is on the same physical hardware as the target, section (7) analyses EC2’s VM placement strategy and explores techniques the attacker can use to achieve reliable co-residence with a target VM.   In the context of a brute force placement attack they note that “even a very naive attack strategy can successfully achieve co-residence against a not-so-small fraction of targets”.  Thus if your target set is wide, brute force turns out to be a viable strategy.  For individual or small sets of targets, a more efficient strategy is “instance flooding” (spinning up numerous VMs) immediately after the target has booted to “take advantage of the parallel placement locality exhibited by the EC2 placement algorithms”.  This is where the dynamic nature of cloud comes into play:

But why would we expect that an attacker can launch instances soon after a particular target victim is launched?  Here the dynamic nature of cloud computing plays well into the hands of creative adversaries. Recall that one of the main features of cloud computing is to only run servers when needed. This suggests that servers are often run on instances, terminated when not needed, and later run again.  So for example, an attacker can monitor a server’s state (e.g., via network probing), wait until the instance disappears, and then if it reappears as a new instance, engage in instance flooding. Even more interestingly, an attacker might be able to actively trigger new victim instances due to the use of auto scaling systems. These automatically grow the number of instances used by a service to meet increases in demand. (Examples include scalr ³⁰ and RightGrid ²⁸.  See also ⁶.) We believe clever adversaries can find many other practical realizations of this attack scenario.

With co-residence achieved, section (8) assesses the practicality of side channel attacks in a VM environment.  As you’d expect, a number of possibilities exist, however the holy grail of cryptographic key extraction does not appear plausible (at this time).  One notable quote:

The side channel attacks we report on in the rest of this section are more coarse-grained than those required to extract cryptographic keys. While this means the attacks extract less bits of information, it also means they are more robust and potentially simpler to implement in noisy environments such as EC2.

Side channel attacks discussed include:

• Denial of Service
• Measure cache usage (measure CPU utilisation on the physical machine; or “how busy are their servers?”)
• Load-based co-residence detection (aka detecting co-residence without relying on sending any network probes)
• Estimating traffic rates (sounds harmless but can be used to deduce targets activity patterns, peak trading times for maximal DoS effect etc)
• Keystroke timing attack (remote keystroke monitoring)

As with each of the other sections, the authors suggest potential countermeasures.

Overall the paper makes a very interested read.  There’s no EC2 “0-day”, but that’s not the intent of the paper.  Rather, we are reminded that cloud platforms and technologies do bring some novel attacks that thus far have not really figured in much of the security conversation to date.  We need more of this type of research to better understand what we are getting ourselves into.

On the 23rd June, Amazon quietly rolled out a security fix for an issue originally discussed in the Amazon developer forums. Amazon documentation was revised to reflect the change as follows:

“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”

Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).

What does this mean?

It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only if you manually verify the host SSH key during your initial SSH connection.

OK, but I created my AMI before June 23rd – am I vulnerable?

According to Amazon, yes.  Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI.  The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.

How can I fix my pre-June 23rd AMIs?

Regenerate the SSH host key.  The exact commands will depend on your operating system (hint: ssh-keygen).

Who is to blame?

Either the creators of the original AMI or Amazon – depends how you look at it.  If Amazon created the public AMI then it could be argued they are responsible.  However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).

Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image).   That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?

Just to be clear, its not the fault of SSH – ’secure channels’ require proper key management and the need for unique host keys is well documented.

Are there any mitigating factors?

Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet).  You’ll still want to regenerate the ssh host keys sooner than later.

Is the Amazon environment vulnerable to Man-in-the-middle attacks?

I don’t know.  But that isn’t the real question – is the path between you and your AMI immune to MITM attacks and the answer is most definitely no.  If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-).  From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).

Why Didn’t Amazon Tell Me I’m Vulnerable?  They know from their logs what AMIs I use!

Didn’t they?  Whoops – naughty Amazon :P.

But seriously, Amazon are not responsible for the configuration of the public AMIs you use.  Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.

Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).

Not that I know of.   Right now you have to search forum posts and monitor documentation updates – which is time consuming and makes it easy to miss something.  I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management).   In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security  (for some customers, if they don’t “see it”, it doesn’t “exist”).

A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.