Cloud Security

Despite what the cynics say, Twitter is  extremely valuable to track and participate in conversations about cloud computing/security, aswell as information security in general.

For those of us that didn’t make it to Black Hat USA/Defcon, the infosec twitter community gave us the next best thing - a running commentary of the presentations - many of which feature cutting edge security research.

I was particularly interested in following the Sensepost presentation called ‘Clobbering the Cloud’. From the write-up:

Cloud Computing dominates the headlines these days but like most paradigm changes this introduces new risks and new opportunities for us to consider. Some deep technical research has gone into the underlying technologies (like Virtualization) but to some extent this serves only to muddy the waters when considering the overall threat landscape. During this talk, SensePost will attempt to separate fact from fiction while walking through several real-world attacks on “the cloud.” The talk will focus both on attacks against the cloud and on using these platforms as attack tools for general Internet mayhem. For purposes of demonstration we will focus most of our demos and attacks against the big players…

In reverse order, check out the tweets from @GphreakX who was at BH and kindly tweeting proceedings:

Some interesting tweets there for sure!  Hopefully this has whet your appetite for the upcoming cloudsecurity.org interview with Haroon and his Sensepost team…stay tuned.

Google just made three big announcements that reveals more about their cloud strategy, security & positioning with enterprises.

Google Chrome Operating System

Perhaps the biggest news is their plan to create a new operating system, based on the Linux kernel, running on X86 and ARM chipsets and targeted at the Netbook/Laptop/Desktop user:

“Google Chrome OS is an open source, lightweight operating system that will initially be targeted at netbooks. Later this year we will open-source its code, and netbooks running Google Chrome OS will be available for consumers in the second half of 2010.”

Talking of their goals:

“Speed, simplicity and security are the key aspects of Google Chrome OS. We’re designing the OS to be fast and lightweight, to start up and get you onto the web in a few seconds. The user interface is minimal to stay out of your way, and most of the user experience takes place on th web.”

And starting from a clean slate (and an obligatory swipe at Microsoft):

“And as we did for the Google Chrome browser, we are going back to the basics and completely redesigning the underlying security architecture of the OS so that users don’t have to deal with viruses, malware and security updates. It should just work.

<snip>

The software architecture is simple — Google Chrome running within a new windowing system on top of a Linux kernel. For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. And of course, these apps will run not only on Google Chrome OS, but on any standards-based browser on Windows, Mac and Linux thereby giving developers the largest user base of any platform.

Google Chrome OS is a new project, separate from Android. Android was designed from the beginning to work across a variety of devices from phones to set-top boxes to netbooks. Google Chrome OS is being created for people who spend most of their time on the web, and is being designed to power computers ranging from small netbooks to full-size desktop systems. While there are areas where Google Chrome OS and Android overlap, we believe choice will drive innovation for the benefit of everyone, including Google.”

Wow, pretty big announcement with lots of potential market implications.  One way to look at this is they just described a system with “embedded OS” properties running as a mainstream desktop OS with services delivered via the web instead of relying on locally hosted applications.  I suppose in some ways this should come as no real surprise as it is entirely in-line with their cloud based strategy.

Whilst the target market would appear to be consumers, I can see enterprises jumping at a thin OS “that just works”.  Ultimately, this is moving us closer to  an age of disposable computing - low cost devices with low entry software footprints.

Organisations are keen to embrace smaller footprint client computers to cut costs and if the underlying hardware offers enterprise demanded features like full HD encryption (to protect that cached Cloud content), I could see enterprises taking a serious interest.

Do we *really* want to run the dozen endpoint agents we have today for configuration management, NAC, AV, HIPS (pah!) and bear all the costs they bring? With a static client, you won’t need many of these features.  From a security point of view, this could be a very good thing - no AV headaches, significantly less attack surface (enterprise apps often demonstrate “brittle” security) and less PII to lose.

To deliver on a low-update OS, they will need to ship a subset of the Linux kernel that is considered “mature”, otherwise their users will be back on the  “patch treadmill” - which is something they explicitly state they are trying to avoid.

I find it interesting they are designing a new windowing system when there are so many options available today (some with decent security too).  I suspect this is to take advantage of advances in graphical chipsets.  Perhaps they see this as a chance to boost Chrome browser page rendering speed even further.

Perhaps the more fundamental question is whether we want Google owning the last bastion - our desktops.

This brings us to the Chrome browser itself and associated technologies.

Google Native Client

Back in February, Google kicked off a security contest for a “research project” called Google Native Client (NaCl).  First a quick recap on Native Client:

“Native Client is an open-source research technology for running x86 native code in web applications, with the goal of maintaining the browser neutrality, OS portability, and safety that people expect from web apps. We’ve released this project at an early, research stage to get feedback from the security and broader open-source communities. We believe that Native Client technology will someday help web developers to create richer and more dynamic browser-based applications. ”

This is Google’s ambitious attempt to provide a high-speed, browser hosted application alternative to Java or Flash. To do this securely, they designed a new security architecture and NaCl is the implementation.

Announcing the security contest, Henry Bridge from Google wrote:

“Exploits, bugs, vulnerabilities, security holes — for most programmers these terms are synonymous with fire drills and coding all-nighters. However, for the next 10 weeks, the Native Client team is inviting you to bring them on! We’re challenging you to find security exploits in Native Client.”

The judges, led by respected academic Ed Felton (Princeton), assessed the vulnerablities reported by each of the 600 participants based on “a) Quality (Severity, Scope, Reliability and Style) and b) Quantity”.  Participants were limited to reporting on 10 bugs (Google claimed this was to avoid wasting the judges time).

Mark Dowd and Ben Hawkes won the contest, finding the bulk of the best bugs. Mark Dowd is well known in the security community - most often described as a humble genius (or a robot sent back in time :).  I followed along at home and it was great fun reading the bug descriptions as the competition progressed.  As this was a new security design, there were some unique vulnerabilities discovered along with novel exploit avenues.  Despite all the implementation snafus, Google is taking comfort that no underlying architectural weaknesses were found.

“This contest helped us discover implementation errors in Native Client and some areas of our codebase we need to spend more time reviewing. More importantly, that no major architectural flaws were found provides evidence that Native Client can be made safe enough for widespread use. Toward that end, we’re implementing additional security measures, such as an outer sandbox…”

In other posts, Google has indicated the plan to bundle NaCl with the browser, rather than offer as a end-user download.  There is some way to go before this happens, and the security contest is just one step on the journey before NaCl goes live.   The NaCl team also submitted a detailed technical design paper to the IEEE 2009 Symposium on Security and Privacy.  If anyone knows anything on the outcome of the peer review, please leave a comment.

Overall, it has to be said that the NaCl team at Google is doing a solid job trying to flush out security issues before “Primetime”.

Having said that, not all observers agree the architecture is a step in the right direction.  Noted reverse engineer Halvar Flake responded to a post by Dave Aitel on the Daily Dave mailing list remarking that:

“The real beauty in NaCl is that it is certain to defeat DEP [Ed: Data Execution Prevention is an hardware and/or software enabled chipset technology design to throw an exception when an attempt is made to pass off data pages as code pages]. Not that DEP is much of an obstacle in browsers these days, but still. It’ll also almost certainly allow ASLR bypass.

Everyone who has even been to one of my classes has been tortured with the analogy that “writing an exploit is like trying to build a chair out of a number of random parts from the IKEA warehouse: Nothing ever fits, but the more pieces you have, the better your odds of success are.

The power to first execute Javascript to perform [Ed: memory] allocations/dealloctions, coupled with the ability to load arbitrary code into the address space that is only verified under alignment assumptions violated as soon as you can perform a control hijack, does look like a jar of superglue to me. And when you have a sufficiently large jar of superglue, you can essentially build a chair out of wood shavings.”

The point that Halvar is making is that the exploit coder has certain advantages when it comes to exploiting browser based weaknesses. Couple this with the very feature that NaCl introduces - loading Internet hosted native code - and any single implementation weakness makes way for reliable exploitation potential bypassing CPU anti-exploitation features.  This kind of dialogue is very constructive and I look forward to seeing how the thinking around NaCl develops.

Google Apps: Beta Out, Enterprise Features In

Back to the Google announcements, and the day finally came Google dropped ‘Beta’ from Google Apps, Gmail, Google Calendar, Google Docs and Google Talk. This is clearly to please enterprise folks who take the traditional interpretation of “beta == buggy”.  Its hard for a CIO to get buy-in with their own org to adopt a hosted service that has those 4 letters staring back at them (even if they agree with Google’s definition of “beta”.  “Premium beta” anyone? ;-).

Google also added email delegation, retention, DR features to Google Apps, along with “special handling of business users’ data in our data center operations.” If anyone has any details on that last point, do share.  Google is in catch-up mode and ticking the right boxes.

All in all, this was a big day for Google and their evolving Cloud strategy, enterprise security people should take note…

Much has been said about public IaaS providers that expressly forbid customers from running network scans against their cloud hosted infrastructure.  Failure to comply with the Terms of Service can result in account suspension or termination (ouch!).  This post is my attempt to suggest a way forward.  I welcome your feedback…

As has been noted before, a blanket ban on legitimate scanning activity by customers of their own infrastructure (whether outsourced or not) undermines security assurance processes and can make regulatory compliance impossible; e.g. PCI DSS mandates network vulnerability scanning as a control.

Vulnerability scanning is a stalwart practice of the Information Security community. Enterprises invest considerable time and money developing vulnerability management programs to help assess IT security risk across applications and infrastructure. Specifically, vulnerability scanners help identify potential security weaknesses at scale; e.g. missing patches, default passwords, coding or configuration weaknesses.

Vulnerability scanning is front of mind for Internet exposed or partner connected infrastructure. However, when said infrastructure is owned and/or operated by a service provider, some of the existing challenges associated with vulnerability scanning are magnified:

• Scans can cause outages.  This can happen if the scanning policy includes Denial of Service checks or the scanning engine is configured with “aggressive” settings; e.g. connection entries in firewall state tables get exhausted.  Its also possible for scans to tickle obscure bugs in the target - or devices enroute to the target.  Even without a full-on outage, poorly configured scans can still negatively impact performance or availability for other customers of shared infrastructure.
• Identifying unauthorised scans. Without a trusted, robust process for “blessing” or approving source IP addresses of customer scan engines, service providers cannot distinguish legitimate scans from scans with the evil bit set.  Sure, they can use whois to determine source network ownership but even if the scan originates from a customer owned network, this does not necessarily mean it is authorised!  Given this, many providers take the stance that all scans are treated as hostile unless pre-agreed.
• Scanning may trigger automated or manual actions by the provider. A common automated response from a provider is to apply traffic shaping to slow down the scan, or simply block the client IP address via an ACL update.  This can lead to false negatives; i.e. vulnerabilities present are not discovered as the scanner IP was automagically identified as a noisy vulnerability scanner and auto-throttled/blocked.  Even half smart attackers can quickly deduces the presence of auto-response mechanisms (”huh, no response now”) so either switches to slow probes from multiple sources or goes for gold with a one-shot exploit.

Enterprise customers on dedicated infrastructure at Tier 1 web hosting providers will either contract the hosting company (or their security partner) to perform vulnerability scans or do it themselves.  Either way, for scanning to happen, agreement will need to be reached on scan scope, types of scans to be run (scanning tools & policies), time windows and source IP addresses used.  Beyond that are the process issues of how results will be communicated, integration with ticketing systems etc.

The provider will limit the scan scope to the dedicated infrastructure allocated to the customer - the scanning of shared infrastructure by the customer is generally a ‘no no’.  This, along with management networks will be scanned by the provider to meet customer compliance mandates or security policies.

With Cloud “Infrastructure as a Service” providers, things get a little more complicated.

• A cloud is multi-tenant; i.e. the cloud platform is shared to multiple customers through software abstraction.  The provider will naturally be concerned with the impact of any scanning activity, particularly if it causes any SLA violations.
• Further, cloud customers can spin up infrastructure on demand.  New virtual servers can be  brought to life automagically to handle increased load.  This increased infrastructure footprint is still subject to the same compliance mandates though; i.e. it must be scanned within some time period of its appearance.  Even if spinning up copies of “known good/secure” virtual machine (VM), you still need to scan them.   New vulnerablities are published all the time, along with corresponding vulnerability checks - hence the need for both regular scans and representative scans.  Further, vulnerbility scanning isn’t just testing the VM, its also helping you verify the security controls outside the VM that are designed to protect it; e.g. a providers’ software firewall.  Picking and choosing which pieces of your hosted infrastructure to scan is a slippery slope to selective exposure if not handled with care.
• Finally, we shouldn’t discount the “Clouding around” factor.  Credit card payments for “instant on” infrastructure changes the dynamic between cloud consumer and cloud provider.  Similar to low end, consumer oriented shared hosting before it, you may never speak with, let alone meet, an employee of your provider before you use their services.  There simply isn’t a conversation about scanning (the “conversation” today is a monologue found in the Terms of Service).  Plus, if the provider fails to meet your needs, you can drop them at a moments notice and switch to another (Cloud baggage permitting…).  In other words, its either not possible, or not convenient to call up your provider to agree the principle and logistics of scanning the services they host on your behalf.  Enterprise customers - or at least their security teams - will be wanting that conversation and can likely strike a deal with a modified ToS to allow scanning of some sort but this seems unncessarily exclusionist to me.

We can address these issues through a mix of provider open-mindedness, policy, process, technology and contract.

For cloud providers to attract certain customers, they may need to soften their policy on vulnerability scanning.  Taking a hardline “no” stance precludes some workloads from ever entering the cloudosphere (with bigger consequences for enterprises seeking a strategic cloud partner).  A preferred scenario has the cloud provider showing some understanding of enterprise prospects assurance needs and defining scanning parameters acceptable to their own operations risk tolerance.

Scanning is not an “unknown” risk, rather its a very well understood activity with quantifiable elements (packet rate, state table usage etc).  Normal rate limiting could be temporarily or permanently loosened for customer approved IP addresses to enable scans against a customers cloud IP addresses (not API endpoints or cloud providers websites!) to complete in a reasonable time window.  Besides, Internet systems are scanned, probed and attacked constantly by script kiddies, Internet surveyors and an assortment of bots and other lifeforms.  So the bad guys get to scan because they don’t care and yet the customer, who wants to do the “right thing”, is not allowed to.  Is that rational?

Assuming a cloud provider with a more measured approach towards vulnerability scanning of customer cloud infrastructure, we now need a simple, mutually trusted mechanism to agree scan sources, rate limits etc.  Something like an “ScanAuth” (Scan Authorize) API call offered by cloud providers that a customer can call with parameters for conveying source IP address(es) that will perform the scanning, and optionally a subset of their Cloud hosted IP addresses, scan start time and/or duration. This request would be signed by the customers API secret/private key as per other privileged API calls. The provider receiving the request can rely on the digital signature as proof that a scan is authorised with the associated parameters. After the provider has processed the scan authorisation request, the provider could return a status code approving or denying the request (with a possible reason code to allow resubmission with more acceptable parameters).  This response can optionally include rate limits which the customer can use to tune the intensity of their scanner.

The provider can now whitelist the customer provided scanner IP(s) for the duration of the requested scanning window such that active countermeasures like anti-DoS controls are not triggered, resulting in a ‘cleaner’ scan (and hence a more accurate report).

Should the scanning activity exceed any specified limits, or communicate with IP addresses not associated with customer virtual machines, the provider could instantly blacklist the scanning IP or apply traffic shaping.

The bottom line: when everyone is clear on the need, approval process, scan parameters and abuse policy, this can be done with very little fuss.

A “ScanAuth” API call empowers the customer (or their nominated 3rd party) to scan their hosted Cloud infrastructure confident in the knowledge they won’t fall foul of the providers Terms of Service. This avoids a situation where either a customers Cloud services are interrupted by an angry provider (availability fail!) or in the worst case, getting kicked off the Cloud entirely.  Clearly, a lose/lose scenario.

What do you think?

Have you ever performed a security review or audit of a 3rd party hosting provider before your employer signs on the dotted line?  Did you ever “have that moment?”.  It’s that time when exhausted from review fatigue you find yourself banging your head on the desk screaming ‘there must be a faster way’.  Well, you’re not the only one…

The scene goes something like this:

The provider rolls their eyes as yet another customer security team sends in their 500 deeply probing security questions, transmitted in some homegrown template in Word, Excel or $diety forbid, Powerpoint.  The customer security team, naturally suspicious of the provider and irked by managements apparent keenness to outsource the farm, has created the security questionnaire from hell:

• it’s the result of 100 hours of internal team meetings
• it’s gone through 14 drafts, 20 reviewers inboxes, 76 yellow highlighter comment fields and was printed at least 6 times
• it only asks IT security questions (no input from other relevant functions such legal/compliance/audit - HA!)
• it’s laced with a few tricky landmine questions based on potential security issues raised (but not satisfactorily answered) in online forums and provider support forums
• it contains 25 attachments detailing all the company security policies that *must* be followed (huh, Bluetooth policy requirements for a cloud storage provider…interesting)

In the context of cloud providers, they are slammed - a raft of audits in progress right now - with more expected soon.  The provider is experiencing an ADoS (Audit Denial of Service).  Instead of innovating new service offerings (including security!), the talented security professional at the provider is stuck cut and pasting answers from internal cheatsheets to customers questionnaires in the knowledge that the customer likely has no idea how much money it would cost to fulfill some of these security requests.  The sheer number of questions is confusing given that the customer IT team had stated they were only looking to host non-critical, non-sensitive data…

Audits are time consuming, repetitive across customers, costly and generally a motivational drain for everyone involved.  Moreover in the context of Cloud, time consuming audits seriously delays a key benefit of cloud - agility.  Its the “on demand” part of “Infrastructure on demand” that is a primary benefit of cloud.  If the security review process takes 3 months to complete, how much business opportunity has your employer lost?  Don’t like that question?  OK, another one: how much time could you have spent doing something more interesting?

Which leads me to some questions:

• what does the cost/benefit ratio look like of the “questionnaire security review method”? (to be clear, I’m not arguing against the need for security reviews)
• why do we all use different format questionnaires? (note: format)
• why are we asking these questions? (are the bulk of our questions simply an expression of our policy asked in a question format?)
• how many of these questions/policies are predictable and duplicated?  As in, you and I ask some of the same questions…we may differ in the details (e.g. password complexity..eek!) but we both probably ask the same base question even if our thresholds around answers are slightly different.
• what if we were to agree a set of common questions/policy statements?  We don’t all have to subscribe to them, we can pick the ones that reflect our policy…  There could be thousands, you search, pick and mix just like an iTunes playlist (Ed: Genius!)
• for those standard policy questions, could we “digitize” them and express them electronically?  Could the provider host a policy oracle that we could post these questions to?
• for those “uncommon” questions that the providers oracle cannot automagically answer, could we agree a standard way to “ask/transmit” those with some simple agreements about response formats? (um, freetext fields ;-).
• ultimately, could we “digitize” a significant portion of our questions to get near instant answers? (and could we make that multi-lingual…)
• would the provider recognise this as a benefit too?
• would the provider also see the legitimate opportunity this presents to charge for higher assurance services around cloud compute/storage/network based on our policy requirements?  “You want triple cycle, double buffering?  You got it - for an extra 5c per MB”).  Yes, the cost of  your security policies in a pay per drink model are revealed!
• would the provider recognise the opportunity to offer incentives to customers for choosing this low friction path of policy compliance instead of tying up their skilled employees filling out ad-hoc questionnaires?

Is there an existing system/application/protocol whereby I can transmit my policy requirements to a provider, they can respond in real-time with compliance level and any additional costs, with less structured/known requirements responded to by a human (but transmitted the same way)?  In other words, I’m looking for human driven, machine to machine policy exchange/agreement.

I propose that the benefit of quickly ascertaining policy compatibility along with any additional costs involved would reduce the on-ramp to cloud, reduce switching costs, drive a form of policy interoperability and take us closer to where we need to be in the long run: the ability to express security policy for a single unit of compute/storage/network in a cloud.  Ultimately, I want to be able to tie my security policy to the information asset I need to protect and push that to a cloud broker who performs policy reconciliation to determine which of my approved provider(s) can meet my needs without any human intervention (yeah, I can hope ;-).

And before everyone jumps on me and says ‘but the point of an on-site audit/security review is to get assurance that the provider is doing what they claim they are doing” I’d like to point out that policy and assurance are two different things.  Before you and the provider invest time in the optional on-site audit, why not get the bulk of the policy questions out the way in a fast and low cost manner? (i.e. “death to the questoinnaire?”).

If you’re following along thus far, you’ll also see the possibility for trusted 3rd party auditors to digitally ’sign’ individual policy statements made by cloud providers they have audited. That signature could itself reflect the assurance level you need.  This in turn could help drive the nascent cyberinsurance market for cloud…assuming the auditor is open to counterclaims by the insurer ;-).

If you do need to go on-site (and assuming the cloud provider tells you where “on-site” is ;-), you’ll have a list of items the provider categorically stated they do, meaning you can cherry pick the areas where you want to deep dive for assurance.  If upon inspection you find reality does not match stated policy, you can scream bloody murder.  Providers that mislead customers will soon get known.

Thoughts?

In a move I found a tad ‘uncloudlike’, ZDNet reports that SUN UK CTO Wayne Horkan is trying to pull together a UK specific Cloud Security group.

On the one hand I totally understand the need for a nation to protect its own interests - particularly where national critical infrastructure is concerned, but on the other, it “feels” a bit strange that an initiative like this is coming from a vendor with a vested interest in Cloud.

Here’s the quote:

Sun’s UK chief technology officer is working with major British public and private organisations to set up a cross-sector forum to resolve cloud-computing security issues.

Cloud-computing systems could become as important as the UK critical national infrastructure, and they need to be secured in an appropriate manner, Wayne Horkan told ZDNet UK on Thursday. The Sun executive said he is working on setting up the forum alongside organisations such as the CBI, Microsoft and Accenture; government departments such as Berr, Dius and the Treasury; and the government’s chief scientific advisor, Professor John Beddington.

“I’m concerned about the security of the supply,” Horkan said at the Cloud Expo Europe conference in London. “If cloud computing becomes a utility, it’s important to me that the UK as a nation state has good security of supply. It’s important that the UK has the appropriate capability in cloud computing.”

He then goes on to cite privacy concerns.

It’s plain to see that the majority of Cloud offerings are from US based companies.  Nearly every briefing I’m invited to is EST or PST.  In fact, I can’t remember even speaking with a UK Cloud provider.   Of the many media requests for comments, all but one were from the US.

I can’t help smelling fear in this effort. As a Brit, I would love to see a UK group coming together to innovate, support and promote the fledgling UK Cloud industry.  Perhaps that will be one of the goals of the group - if so, I don’t think that is ’security’ specific (unless we are talking security innovation).

Development of UK specific Data Privacy guidance in relation to Cloud should be led and enforced by the Information Commissioners Office.

I also feel this will do little to advance security of the Cloud overall. With the positive news yesterday that the UK based Jericho forum and the Cloud Security Alliance (CSA) have formally agreed to “work together”, isn’t this inward looking approach just fragmenting our efforts?  Why not direct the security talent that would comprise this group towards the CSA or ENISA.

Security is a *global* issue.  I’m struggling to see how country specific cloud security interest groups “fit” when we talk about globally distributed systems.  What next - Cloud UN? ;-).

I don’t disagree with the need to protect supply, but I would much prefer to see the UK government driving an initiative like this as part of their critical infrastructure protection strategy.  A strategy around UK Cloud innovation would be nice too ;-).

Perhaps I am being overly pessimistic or missing something. What do you think of a country specific Cloud security group set up by a technology company? A US based technology company no less… ;-).