Cloud Security

This evening I’m off to Brussels, Belgium to attend the very first Brucon - a 2 day information security conference aimed squarely at those curious about technologies from a security point of view.  I’m scheduled to talk about Cloud Security on the last slot on Saturday before the after-party.   Conscious of the timing and location, I’m employing some less-than-subtle marketing tactics to get “bums on seats” as you’ll see below.

Here’s the abstract:

In a hurry? The short version: learn about cloud security and in the process win a tasty Belgian beer by answering easy questions!

When Amazon CEO Jeff Bezos was photographed standing in front of a vintage 1890s electric generator, it was widely assumed he was paying homage to Nick Carr’s “electric generator” metaphor of utility computing. This was understandable, but quite wrong. Reminiscent of the Bruce Lee movie where the student is chided for failing to look “out there” instead of staring at his own hand, the cloud commentators failed to notice his surroundings. Bezos — and the electric generator — were standing in the middle of a Belgium Brewery! This will be the starting point of our journey through Cloud Security using a fuller flavour metaphor: Belgian beer.

In this presentation I will cover:

• why talking about “cloud” is akin to walking into a Belgian bar and asking for “beer”
• the common cloud architectures and their implications for you - the security dude
• what the beer brewing Trappist Monks can teach us about cloud security
• attacking clouds (aka getting free beer)
• dealing with the hangover: cloud incident response & forensics

I plan to post the presentation online when I get back.  If you will be there, do say hello.

That’s right…you thought you couldn’t get enough Cloud Security…well, now you can hear myself and co-host Chris Hoff of Rational Survivability discuss recent cloudsec happenings.

Cloud Security Podcast Episode 1

[If you don't see the player above, turn on Javascript]

Brief show notes:

• Introductions
• Quick recap of what we mean by ‘Cloud Computing’
• Recent news & events (with a focus on FUD)
• Groups developing cloud security guidance: Cloud Security Alliance, Enisa, Jericho
• Wrap-up

Full show notes

As this was our first foray into making our own podcast, we’re seeking your feedback (we know about the audio drop outs).

Tell us what you think…

P.S Submitting to iTunes shortly

Despite what the cynics say, Twitter is  extremely valuable to track and participate in conversations about cloud computing/security, aswell as information security in general.

For those of us that didn’t make it to Black Hat USA/Defcon, the infosec twitter community gave us the next best thing - a running commentary of the presentations - many of which feature cutting edge security research.

I was particularly interested in following the Sensepost presentation called ‘Clobbering the Cloud’. From the write-up:

Cloud Computing dominates the headlines these days but like most paradigm changes this introduces new risks and new opportunities for us to consider. Some deep technical research has gone into the underlying technologies (like Virtualization) but to some extent this serves only to muddy the waters when considering the overall threat landscape. During this talk, SensePost will attempt to separate fact from fiction while walking through several real-world attacks on “the cloud.” The talk will focus both on attacks against the cloud and on using these platforms as attack tools for general Internet mayhem. For purposes of demonstration we will focus most of our demos and attacks against the big players…

In reverse order, check out the tweets from @GphreakX who was at BH and kindly tweeting proceedings:

Some interesting tweets there for sure!  Hopefully this has whet your appetite for the upcoming cloudsecurity.org interview with Haroon and his Sensepost team…stay tuned.

Much has been said about public IaaS providers that expressly forbid customers from running network scans against their cloud hosted infrastructure.  Failure to comply with the Terms of Service can result in account suspension or termination (ouch!).  This post is my attempt to suggest a way forward.  I welcome your feedback…

As has been noted before, a blanket ban on legitimate scanning activity by customers of their own infrastructure (whether outsourced or not) undermines security assurance processes and can make regulatory compliance impossible; e.g. PCI DSS mandates network vulnerability scanning as a control.

Vulnerability scanning is a stalwart practice of the Information Security community. Enterprises invest considerable time and money developing vulnerability management programs to help assess IT security risk across applications and infrastructure. Specifically, vulnerability scanners help identify potential security weaknesses at scale; e.g. missing patches, default passwords, coding or configuration weaknesses.

Vulnerability scanning is front of mind for Internet exposed or partner connected infrastructure. However, when said infrastructure is owned and/or operated by a service provider, some of the existing challenges associated with vulnerability scanning are magnified:

• Scans can cause outages.  This can happen if the scanning policy includes Denial of Service checks or the scanning engine is configured with “aggressive” settings; e.g. connection entries in firewall state tables get exhausted.  Its also possible for scans to tickle obscure bugs in the target - or devices enroute to the target.  Even without a full-on outage, poorly configured scans can still negatively impact performance or availability for other customers of shared infrastructure.
• Identifying unauthorised scans. Without a trusted, robust process for “blessing” or approving source IP addresses of customer scan engines, service providers cannot distinguish legitimate scans from scans with the evil bit set.  Sure, they can use whois to determine source network ownership but even if the scan originates from a customer owned network, this does not necessarily mean it is authorised!  Given this, many providers take the stance that all scans are treated as hostile unless pre-agreed.
• Scanning may trigger automated or manual actions by the provider. A common automated response from a provider is to apply traffic shaping to slow down the scan, or simply block the client IP address via an ACL update.  This can lead to false negatives; i.e. vulnerabilities present are not discovered as the scanner IP was automagically identified as a noisy vulnerability scanner and auto-throttled/blocked.  Even half smart attackers can quickly deduces the presence of auto-response mechanisms (”huh, no response now”) so either switches to slow probes from multiple sources or goes for gold with a one-shot exploit.

Enterprise customers on dedicated infrastructure at Tier 1 web hosting providers will either contract the hosting company (or their security partner) to perform vulnerability scans or do it themselves.  Either way, for scanning to happen, agreement will need to be reached on scan scope, types of scans to be run (scanning tools & policies), time windows and source IP addresses used.  Beyond that are the process issues of how results will be communicated, integration with ticketing systems etc.

The provider will limit the scan scope to the dedicated infrastructure allocated to the customer - the scanning of shared infrastructure by the customer is generally a ‘no no’.  This, along with management networks will be scanned by the provider to meet customer compliance mandates or security policies.

With Cloud “Infrastructure as a Service” providers, things get a little more complicated.

• A cloud is multi-tenant; i.e. the cloud platform is shared to multiple customers through software abstraction.  The provider will naturally be concerned with the impact of any scanning activity, particularly if it causes any SLA violations.
• Further, cloud customers can spin up infrastructure on demand.  New virtual servers can be  brought to life automagically to handle increased load.  This increased infrastructure footprint is still subject to the same compliance mandates though; i.e. it must be scanned within some time period of its appearance.  Even if spinning up copies of “known good/secure” virtual machine (VM), you still need to scan them.   New vulnerablities are published all the time, along with corresponding vulnerability checks - hence the need for both regular scans and representative scans.  Further, vulnerbility scanning isn’t just testing the VM, its also helping you verify the security controls outside the VM that are designed to protect it; e.g. a providers’ software firewall.  Picking and choosing which pieces of your hosted infrastructure to scan is a slippery slope to selective exposure if not handled with care.
• Finally, we shouldn’t discount the “Clouding around” factor.  Credit card payments for “instant on” infrastructure changes the dynamic between cloud consumer and cloud provider.  Similar to low end, consumer oriented shared hosting before it, you may never speak with, let alone meet, an employee of your provider before you use their services.  There simply isn’t a conversation about scanning (the “conversation” today is a monologue found in the Terms of Service).  Plus, if the provider fails to meet your needs, you can drop them at a moments notice and switch to another (Cloud baggage permitting…).  In other words, its either not possible, or not convenient to call up your provider to agree the principle and logistics of scanning the services they host on your behalf.  Enterprise customers - or at least their security teams - will be wanting that conversation and can likely strike a deal with a modified ToS to allow scanning of some sort but this seems unncessarily exclusionist to me.

We can address these issues through a mix of provider open-mindedness, policy, process, technology and contract.

For cloud providers to attract certain customers, they may need to soften their policy on vulnerability scanning.  Taking a hardline “no” stance precludes some workloads from ever entering the cloudosphere (with bigger consequences for enterprises seeking a strategic cloud partner).  A preferred scenario has the cloud provider showing some understanding of enterprise prospects assurance needs and defining scanning parameters acceptable to their own operations risk tolerance.

Scanning is not an “unknown” risk, rather its a very well understood activity with quantifiable elements (packet rate, state table usage etc).  Normal rate limiting could be temporarily or permanently loosened for customer approved IP addresses to enable scans against a customers cloud IP addresses (not API endpoints or cloud providers websites!) to complete in a reasonable time window.  Besides, Internet systems are scanned, probed and attacked constantly by script kiddies, Internet surveyors and an assortment of bots and other lifeforms.  So the bad guys get to scan because they don’t care and yet the customer, who wants to do the “right thing”, is not allowed to.  Is that rational?

Assuming a cloud provider with a more measured approach towards vulnerability scanning of customer cloud infrastructure, we now need a simple, mutually trusted mechanism to agree scan sources, rate limits etc.  Something like an “ScanAuth” (Scan Authorize) API call offered by cloud providers that a customer can call with parameters for conveying source IP address(es) that will perform the scanning, and optionally a subset of their Cloud hosted IP addresses, scan start time and/or duration. This request would be signed by the customers API secret/private key as per other privileged API calls. The provider receiving the request can rely on the digital signature as proof that a scan is authorised with the associated parameters. After the provider has processed the scan authorisation request, the provider could return a status code approving or denying the request (with a possible reason code to allow resubmission with more acceptable parameters).  This response can optionally include rate limits which the customer can use to tune the intensity of their scanner.

The provider can now whitelist the customer provided scanner IP(s) for the duration of the requested scanning window such that active countermeasures like anti-DoS controls are not triggered, resulting in a ‘cleaner’ scan (and hence a more accurate report).

Should the scanning activity exceed any specified limits, or communicate with IP addresses not associated with customer virtual machines, the provider could instantly blacklist the scanning IP or apply traffic shaping.

The bottom line: when everyone is clear on the need, approval process, scan parameters and abuse policy, this can be done with very little fuss.

A “ScanAuth” API call empowers the customer (or their nominated 3rd party) to scan their hosted Cloud infrastructure confident in the knowledge they won’t fall foul of the providers Terms of Service. This avoids a situation where either a customers Cloud services are interrupted by an angry provider (availability fail!) or in the worst case, getting kicked off the Cloud entirely.  Clearly, a lose/lose scenario.

What do you think?

Have you ever performed a security review or audit of a 3rd party hosting provider before your employer signs on the dotted line?  Did you ever “have that moment?”.  It’s that time when exhausted from review fatigue you find yourself banging your head on the desk screaming ‘there must be a faster way’.  Well, you’re not the only one…

The scene goes something like this:

The provider rolls their eyes as yet another customer security team sends in their 500 deeply probing security questions, transmitted in some homegrown template in Word, Excel or $diety forbid, Powerpoint.  The customer security team, naturally suspicious of the provider and irked by managements apparent keenness to outsource the farm, has created the security questionnaire from hell:

• it’s the result of 100 hours of internal team meetings
• it’s gone through 14 drafts, 20 reviewers inboxes, 76 yellow highlighter comment fields and was printed at least 6 times
• it only asks IT security questions (no input from other relevant functions such legal/compliance/audit - HA!)
• it’s laced with a few tricky landmine questions based on potential security issues raised (but not satisfactorily answered) in online forums and provider support forums
• it contains 25 attachments detailing all the company security policies that *must* be followed (huh, Bluetooth policy requirements for a cloud storage provider…interesting)

In the context of cloud providers, they are slammed - a raft of audits in progress right now - with more expected soon.  The provider is experiencing an ADoS (Audit Denial of Service).  Instead of innovating new service offerings (including security!), the talented security professional at the provider is stuck cut and pasting answers from internal cheatsheets to customers questionnaires in the knowledge that the customer likely has no idea how much money it would cost to fulfill some of these security requests.  The sheer number of questions is confusing given that the customer IT team had stated they were only looking to host non-critical, non-sensitive data…

Audits are time consuming, repetitive across customers, costly and generally a motivational drain for everyone involved.  Moreover in the context of Cloud, time consuming audits seriously delays a key benefit of cloud - agility.  Its the “on demand” part of “Infrastructure on demand” that is a primary benefit of cloud.  If the security review process takes 3 months to complete, how much business opportunity has your employer lost?  Don’t like that question?  OK, another one: how much time could you have spent doing something more interesting?

Which leads me to some questions:

• what does the cost/benefit ratio look like of the “questionnaire security review method”? (to be clear, I’m not arguing against the need for security reviews)
• why do we all use different format questionnaires? (note: format)
• why are we asking these questions? (are the bulk of our questions simply an expression of our policy asked in a question format?)
• how many of these questions/policies are predictable and duplicated?  As in, you and I ask some of the same questions…we may differ in the details (e.g. password complexity..eek!) but we both probably ask the same base question even if our thresholds around answers are slightly different.
• what if we were to agree a set of common questions/policy statements?  We don’t all have to subscribe to them, we can pick the ones that reflect our policy…  There could be thousands, you search, pick and mix just like an iTunes playlist (Ed: Genius!)
• for those standard policy questions, could we “digitize” them and express them electronically?  Could the provider host a policy oracle that we could post these questions to?
• for those “uncommon” questions that the providers oracle cannot automagically answer, could we agree a standard way to “ask/transmit” those with some simple agreements about response formats? (um, freetext fields ;-).
• ultimately, could we “digitize” a significant portion of our questions to get near instant answers? (and could we make that multi-lingual…)
• would the provider recognise this as a benefit too?
• would the provider also see the legitimate opportunity this presents to charge for higher assurance services around cloud compute/storage/network based on our policy requirements?  “You want triple cycle, double buffering?  You got it - for an extra 5c per MB”).  Yes, the cost of  your security policies in a pay per drink model are revealed!
• would the provider recognise the opportunity to offer incentives to customers for choosing this low friction path of policy compliance instead of tying up their skilled employees filling out ad-hoc questionnaires?

Is there an existing system/application/protocol whereby I can transmit my policy requirements to a provider, they can respond in real-time with compliance level and any additional costs, with less structured/known requirements responded to by a human (but transmitted the same way)?  In other words, I’m looking for human driven, machine to machine policy exchange/agreement.

I propose that the benefit of quickly ascertaining policy compatibility along with any additional costs involved would reduce the on-ramp to cloud, reduce switching costs, drive a form of policy interoperability and take us closer to where we need to be in the long run: the ability to express security policy for a single unit of compute/storage/network in a cloud.  Ultimately, I want to be able to tie my security policy to the information asset I need to protect and push that to a cloud broker who performs policy reconciliation to determine which of my approved provider(s) can meet my needs without any human intervention (yeah, I can hope ;-).

And before everyone jumps on me and says ‘but the point of an on-site audit/security review is to get assurance that the provider is doing what they claim they are doing” I’d like to point out that policy and assurance are two different things.  Before you and the provider invest time in the optional on-site audit, why not get the bulk of the policy questions out the way in a fast and low cost manner? (i.e. “death to the questoinnaire?”).

If you’re following along thus far, you’ll also see the possibility for trusted 3rd party auditors to digitally ’sign’ individual policy statements made by cloud providers they have audited. That signature could itself reflect the assurance level you need.  This in turn could help drive the nascent cyberinsurance market for cloud…assuming the auditor is open to counterclaims by the insurer ;-).

If you do need to go on-site (and assuming the cloud provider tells you where “on-site” is ;-), you’ll have a list of items the provider categorically stated they do, meaning you can cherry pick the areas where you want to deep dive for assurance.  If upon inspection you find reality does not match stated policy, you can scream bloody murder.  Providers that mislead customers will soon get known.

Thoughts?