Is Your Amazon Machine Image Vulnerable to SSH Spoofing Attacks?
On the 23rd June, Amazon quietly rolled out a security fix for an issue originally discussed in the Amazon developer forums. Amazon documentation was revised to reflect the change as follows:
“Amazon EC2 public AMIs (Amazon Machine Image) generate unique SSH (Secure Shell) host keys each time you launch an instance. This enables you to get the host SSH keys from the console output and verify the host to which you are connecting.”
Important note: SSH host keys enable clients to verify the server identity (”are you really my server?”) and are separate from SSH user keys that allow the user to prove their identity to the server (”he really is Jeff”).
What does this mean?
It means that EC2 instances created from a public AMI after June 23rd have unique SSH host keys and thus are not vulnerable to a man in the middle attack against the SSH protocol, but only *if* you manually verify the host SSH key during your initial SSH connection.
OK, but I created my AMI before June 23rd - am I vulnerable?
According to Amazon, yes. Every EC2 instance copied from a public AMI will have the same SSH host keys as the original AMI. The only exception to this is if the original AMI creator spotted this problem and used a hook to force SSH host key regeneration upon first boot. This means that an attacker who say, uses a DNS cache poisoning attack, can intercept the communication between your SSH client and your AMI.
How can I fix my pre-June 23rd AMIs?
Regenerate the SSH host key. The exact commands will depend on your operating system (hint: ssh-keygen).
Who is to blame?
Either the creators of the original AMI or Amazon - depends how you look at it. If Amazon created the public AMI then it could be argued they are responsible. However, anyone can submit a public AMI and Amazon makes no guarantee they are fit for use (Amazon do review the AMI listing according to their documentation).
Amazon can in fact make the argument they are acting in the interests of their users by implementing a shared solution to key regeneration (rather than requiring each user to manually regenerate the ssh host keys after booting an image). That’s fine going forward but what of potential exposure to customers using the pre-June 23rd public AMI copies?
Just to be clear, its not the fault of SSH - ’secure channels’ require proper key management and the need for unique host keys is well documented.
Are there any mitigating factors?
Yes, if you have used security groups to limit SSH access to your AMI from IP ranges you trust (rather than the entire Internet). You’ll still want to regenerate the ssh host keys sooner than later.
Is the Amazon environment vulnerable to Man-in-the-middle attacks?
I don’t know. But that isn’t the real question - is the path between you and your AMI immune to MITM attacks and the answer is most definitely no. If SSH on your AMI is only accessible from another AMI then its a fair question but its unlikely Amazon are going to show you their network diagrams ;-). From experience performing MITM attacks, I would assume most networks are vulnerable (one of the reasons why we use SSH).
Why Didn’t Amazon Tell Me I’m Vulnerable? They know from their logs what AMIs I use!
Didn’t they? Whoops - naughty Amazon :P.
But seriously, Amazon are not responsible for the configuration of the public AMIs you use. Its important not to confuse the AMI selection and cloning mechanism that Amazon provides, with the content of an AMI itself.
Does Amazon have a mailing list for customers to learn about new security problems (even if its not Amazon’s fault).
Not that I know of. Right now you have to search forum posts and monitor documentation updates - which is time consuming and makes it easy to miss something. I also can’t find an area on the AWS website where they collect security related items together (e.g. best practices, advisories, key management). In my view, this is a shame as it probably undermines the effort that Amazon are putting into their security (for some customers, if they don’t “see it”, it doesn’t “exist”).
A ‘Security’ link on the main AWS homepage pointing to those resources would go a long way to improving the visibility of the AWS security related information.
If you are curious about Cloud Computing and security, don’t miss out on future posts: subscribe by RSS or subscribe by email.
Very interesting article, Craig, thank you for sharing. I am not a security expert, but there are three thoughts that I want to add.
#1 How can I fix my pre-June 23red AMIs?
Is this possible at all? If the attacker has already poisoned your DNS cache and led you to a compromised AMI, then you will only change the host key on the wrong host.
#2 Why doesn’t Amazon just give us the public IP on AMI launch? Or is this already possible and I am not aware of it? Then DNS poisoning would be useless, just log in with the IP.
#3 Is it secure to use public AMIs at all? You always have to trust the person who bundled it. In case of Amazon I am quite sure that they don’t want to trick me. There are also a few other contributors, like Eric Hammond, whom I trust. But this is already fuzzy, who should I trust?
@Markus Klems:
Thanks for the comment. My answers:
#1: With Amazon EC2 you can login directly to the console, rather than going via SSH
#2: There is a public IP given at launch if memory serves. However MITM attacks go beyond DNS cache poisoning. For example; ARP spoofing can be used to redirect traffic. This attack happens at layer 2 - the attacker has to be on the same network as the victim (or break into a host on the same network) - and thus works even if an IP is specified. The bottom line is if you’re AMI has already been spoofed then the only guaranteed way out is to create a new AMI from a trusted AMI. How will you know if you’ve been spoofed? Well, there would be some temporal forensic evidence but it really depends on the skillset of the attacker (big muddy footprints vs. invisible). But how many people can realistically check for those signs? Very few and I don’t think its reasonable to expect people to be able to do this.
#3: “secure” is a relative term :-). Is it reasonable to use a public AMI? I think it depends on what you want to use it for. If you are just experimenting, don’t plan on loading any sensitive data then its a great way to get started. But would I use it for a production service collecting customer data? Probably not. But as you say, its about trust - if I trust the person that built the machine and am willing to stake my reputation on it then that may be sufficient (for unregulated stuff). I agree that Amazon won’t want to trick you but that suggests intention rather than accidental (human error - poorly configured systems - is perhaps a bigger risk!). I predict we’ll see ’security certified AMI images’ or similar at some point. Security companies won’t be able to resist ;-).
In VWS clouds, the VM can be bootstrapped with a session key by the infrastructure. This session key is used to (among other things) report the generated public SSHd host key through authenticated channels — all the way back to the remote client. This allows the local known_hosts file to be adjusted before the first login.
See http://workspace.globus.org/clouds/clusters.html#secgap