Cloud Security

Google just made three big announcements that reveals more about their cloud strategy, security & positioning with enterprises.

Google Chrome Operating System

Perhaps the biggest news is their plan to create a new operating system, based on the Linux kernel, running on X86 and ARM chipsets and targeted at the Netbook/Laptop/Desktop user:

“Google Chrome OS is an open source, lightweight operating system that will initially be targeted at netbooks. Later this year we will open-source its code, and netbooks running Google Chrome OS will be available for consumers in the second half of 2010.”

Talking of their goals:

“Speed, simplicity and security are the key aspects of Google Chrome OS. We’re designing the OS to be fast and lightweight, to start up and get you onto the web in a few seconds. The user interface is minimal to stay out of your way, and most of the user experience takes place on th web.”

And starting from a clean slate (and an obligatory swipe at Microsoft):

“And as we did for the Google Chrome browser, we are going back to the basics and completely redesigning the underlying security architecture of the OS so that users don’t have to deal with viruses, malware and security updates. It should just work.

<snip>

The software architecture is simple — Google Chrome running within a new windowing system on top of a Linux kernel. For application developers, the web is the platform. All web-based applications will automatically work and new applications can be written using your favorite web technologies. And of course, these apps will run not only on Google Chrome OS, but on any standards-based browser on Windows, Mac and Linux thereby giving developers the largest user base of any platform.

Google Chrome OS is a new project, separate from Android. Android was designed from the beginning to work across a variety of devices from phones to set-top boxes to netbooks. Google Chrome OS is being created for people who spend most of their time on the web, and is being designed to power computers ranging from small netbooks to full-size desktop systems. While there are areas where Google Chrome OS and Android overlap, we believe choice will drive innovation for the benefit of everyone, including Google.”

Wow, pretty big announcement with lots of potential market implications.  One way to look at this is they just described a system with “embedded OS” properties running as a mainstream desktop OS with services delivered via the web instead of relying on locally hosted applications.  I suppose in some ways this should come as no real surprise as it is entirely in-line with their cloud based strategy.

Whilst the target market would appear to be consumers, I can see enterprises jumping at a thin OS “that just works”.  Ultimately, this is moving us closer to  an age of disposable computing - low cost devices with low entry software footprints.

Organisations are keen to embrace smaller footprint client computers to cut costs and if the underlying hardware offers enterprise demanded features like full HD encryption (to protect that cached Cloud content), I could see enterprises taking a serious interest.

Do we *really* want to run the dozen endpoint agents we have today for configuration management, NAC, AV, HIPS (pah!) and bear all the costs they bring? With a static client, you won’t need many of these features.  From a security point of view, this could be a very good thing - no AV headaches, significantly less attack surface (enterprise apps often demonstrate “brittle” security) and less PII to lose.

To deliver on a low-update OS, they will need to ship a subset of the Linux kernel that is considered “mature”, otherwise their users will be back on the  “patch treadmill” - which is something they explicitly state they are trying to avoid.

I find it interesting they are designing a new windowing system when there are so many options available today (some with decent security too).  I suspect this is to take advantage of advances in graphical chipsets.  Perhaps they see this as a chance to boost Chrome browser page rendering speed even further.

Perhaps the more fundamental question is whether we want Google owning the last bastion - our desktops.

This brings us to the Chrome browser itself and associated technologies.

Google Native Client

Back in February, Google kicked off a security contest for a “research project” called Google Native Client (NaCl).  First a quick recap on Native Client:

“Native Client is an open-source research technology for running x86 native code in web applications, with the goal of maintaining the browser neutrality, OS portability, and safety that people expect from web apps. We’ve released this project at an early, research stage to get feedback from the security and broader open-source communities. We believe that Native Client technology will someday help web developers to create richer and more dynamic browser-based applications. ”

This is Google’s ambitious attempt to provide a high-speed, browser hosted application alternative to Java or Flash. To do this securely, they designed a new security architecture and NaCl is the implementation.

Announcing the security contest, Henry Bridge from Google wrote:

“Exploits, bugs, vulnerabilities, security holes — for most programmers these terms are synonymous with fire drills and coding all-nighters. However, for the next 10 weeks, the Native Client team is inviting you to bring them on! We’re challenging you to find security exploits in Native Client.”

The judges, led by respected academic Ed Felton (Princeton), assessed the vulnerablities reported by each of the 600 participants based on “a) Quality (Severity, Scope, Reliability and Style) and b) Quantity”.  Participants were limited to reporting on 10 bugs (Google claimed this was to avoid wasting the judges time).

Mark Dowd and Ben Hawkes won the contest, finding the bulk of the best bugs. Mark Dowd is well known in the security community - most often described as a humble genius (or a robot sent back in time :).  I followed along at home and it was great fun reading the bug descriptions as the competition progressed.  As this was a new security design, there were some unique vulnerabilities discovered along with novel exploit avenues.  Despite all the implementation snafus, Google is taking comfort that no underlying architectural weaknesses were found.

“This contest helped us discover implementation errors in Native Client and some areas of our codebase we need to spend more time reviewing. More importantly, that no major architectural flaws were found provides evidence that Native Client can be made safe enough for widespread use. Toward that end, we’re implementing additional security measures, such as an outer sandbox…”

In other posts, Google has indicated the plan to bundle NaCl with the browser, rather than offer as a end-user download.  There is some way to go before this happens, and the security contest is just one step on the journey before NaCl goes live.   The NaCl team also submitted a detailed technical design paper to the IEEE 2009 Symposium on Security and Privacy.  If anyone knows anything on the outcome of the peer review, please leave a comment.

Overall, it has to be said that the NaCl team at Google is doing a solid job trying to flush out security issues before “Primetime”.

Having said that, not all observers agree the architecture is a step in the right direction.  Noted reverse engineer Halvar Flake responded to a post by Dave Aitel on the Daily Dave mailing list remarking that:

“The real beauty in NaCl is that it is certain to defeat DEP [Ed: Data Execution Prevention is an hardware and/or software enabled chipset technology design to throw an exception when an attempt is made to pass off data pages as code pages]. Not that DEP is much of an obstacle in browsers these days, but still. It’ll also almost certainly allow ASLR bypass.

Everyone who has even been to one of my classes has been tortured with the analogy that “writing an exploit is like trying to build a chair out of a number of random parts from the IKEA warehouse: Nothing ever fits, but the more pieces you have, the better your odds of success are.

The power to first execute Javascript to perform [Ed: memory] allocations/dealloctions, coupled with the ability to load arbitrary code into the address space that is only verified under alignment assumptions violated as soon as you can perform a control hijack, does look like a jar of superglue to me. And when you have a sufficiently large jar of superglue, you can essentially build a chair out of wood shavings.”

The point that Halvar is making is that the exploit coder has certain advantages when it comes to exploiting browser based weaknesses. Couple this with the very feature that NaCl introduces - loading Internet hosted native code - and any single implementation weakness makes way for reliable exploitation potential bypassing CPU anti-exploitation features.  This kind of dialogue is very constructive and I look forward to seeing how the thinking around NaCl develops.

Google Apps: Beta Out, Enterprise Features In

Back to the Google announcements, and the day finally came Google dropped ‘Beta’ from Google Apps, Gmail, Google Calendar, Google Docs and Google Talk. This is clearly to please enterprise folks who take the traditional interpretation of “beta == buggy”.  Its hard for a CIO to get buy-in with their own org to adopt a hosted service that has those 4 letters staring back at them (even if they agree with Google’s definition of “beta”.  “Premium beta” anyone? ;-).

Google also added email delegation, retention, DR features to Google Apps, along with “special handling of business users’ data in our data center operations.” If anyone has any details on that last point, do share.  Google is in catch-up mode and ticking the right boxes.

All in all, this was a big day for Google and their evolving Cloud strategy, enterprise security people should take note…

The Electronic Privacy Information Center (EPIC) has lodged a formal complaint about Google to the US Federal Trade Commission (FTC), insisting that they investigate the adequacy and sufficiency of Googles privacy and security safeguards.  EPIC is also seeking changes to Googles Term of Service and a suspension of Googles Cloud Computing services until ’safeguards’ are verifiably established.  Finally, they want the FTC to “compel Google to contribute 5,000,000USD to a public fund that will help support research concerning privacy enhancing technologies…”.

EPIC forwards this complaint on 3 primary fronts:

• the specific ways that Google represents their security controls to consumers (yet disclaims all responsibilities in the Terms of Service)
• the “harm” caused by the recent Google Docs privacy breach
• the claim that Google has “inadequate security”

Secondary arguments include citing a number of other, older vulnerabilities in Google online services and referencing some significant privacy breaches where the FTC acted before.  In my view, these are distractions and inconsistent with the primary argument.  The call for Google to pay 5 million dollars is poorly framed, seemingly an afterthought and potentially devisive.  I suspect EPIC will have lost the goodwill of privacy moderates by making such a demand.  Had they just dropped the number and left the call for a fund, it might have made it more palatable.

Given the complaint is 15 pages long, there is plenty to comment on.  For the sake of brevity, lets contain our analysis to the primary arguments, introduce a potential curveball and go “one step beyond” to examine the implications for Google users should the FTC rule in EPICs favour.

What Google Says About Security

EPIC highlights two specific security claims made by Google.

On the Google Docs homepage

Getting to know Google Docs> Saving your presentation

The complaint then goes on to suggest that Google “encourages users to add personal information to their documents and spreadsheets” and repeats the statement made by Google that “your data is private, unless you grant access to others and/or publish your information”.

Having built their primary argument based on public statements made on Google online properties, they bring out the Google Terms of Service which states in Section 14.1 that the services are provided “as is”, with no warranty and that Google does “not represent or warrant” that [14.2 B] “your use of the Services will be uninterrupted, timely, secure or free from error”.  Section 15 states that Google will not be liable for losses.

The Harm Caused By The Google Docs Privacy Breach

EPIC then attempts to link the Google Docs privacy breach with harm experienced by Google users:

Curious.  2 sentences in a 15 page report where EPIC could have firmly established the ‘harm’ case.  No examples, no quantification, no impact analysis. Perhaps EPIC is playing its hand carefully and is readying a parade of impacted users who can demonstrate they were “harmed” by the privacy snafu.  Failing that, it would mean they have built their case on the morality of a software privacy bug at a popular online service and ultimately, an industry wide disparity between the big print a company uses to market their services (and software!) as trustworthy and the small print where the lawyers spell out the case for the defense.

The Claim That Google has Inadequate Security

The third and final primary argument.  Skipping past the reminder from EPIC to the FTC that they acted in response to other privacy breaches, EPIC goes on to state that Google’s “inadequate security is an unfair business practice” and that Google’s “Inadequate Security” is a deceptive trade practice”.   They argue that the Google Docs privacy breach was a result of inadequate security practices and that Google:

• encourages people to share “sensitive” documentation in promoting their services
• “knew that Cloud Computing Services are susceptible to data breaches”
• “knew that disclosure of personal user data could cause substantial injury to customers”
• was “aware that commonsense security measures, including storing user data in encrypted form, rather than in clear text, could reduce the likelihood and extent of consumer injury”
• “created an unnecessary risk to users’ data by employing unreasonable security practices, including the storage and transmission of personal information on its computer network in clear text”

What I find fascinating about this is that EPIC is drawing a significant conclusion about Google’s security practices based on the fact that Google doesn’t take “commonsense” security measures. In other words, because Google hasn’t implemented a PKI and DRM for document sharing in a (for many) free service, Google is somehow employing unreasonable security practices (!).

This just strikes me as really unreasonable and wholly unrealistic.  If the FTC mandated those level of security protections to “qualify” for accepting data that consumers choose to put in the Cloud, you can say goodnight to *all* of the popular Web 2.0 services.

With Google thoroughly chastised, they draw the following “big picture” conclusion:

First Google, now the Cloud Computing as a whole - I’d better change my domain name fast! ;-).

A Potential Curveball

After I posted my thoughts on how Google Security responded to the Google Docs sharing problem I was contacted by a Google Docs user who stated that he reported a sharing problem to Google in January. He discovered a large number of documents shared with over a hundred people (he gave specific numbers that I’m intentionally not quoting to protect his privacy).  He states he called Google Tech support who initiated a support case.

Assuming this is true (and based on his note, I have no reason to doubt it) it means either:

• Google Docs has suffered another sharing problem that was quietly fixed (no notification?)
• If this is the same sharing problem, it means at least someone in Google knew about it from late January which completely changes how their responsiveness to dealing with this problem will be perceived.

What If EPIC Gets Their Way

If the FTC went as far as forcing Google to suspend its services, we will witness the largest Denial of Legitimate Service (DoLS) attack in history.

Can you imagine how that would play out?  I suspect it would also be the worst PR disaster of all time for EPIC as Google users turn on them in their droves…

In their concern for privacy, one part of security that EPIC seems to have forgotten is availability and the Cloud is all about that.

Baynote just got shamefaced with the discovery of a basic Cross Site Scripting vulnerability in their ‘Social Search’ SaaS offering. Although – as seems to be the trend – you won’t find this out from reading their blog, press releases or other public areas of their website.  Instead, you learn of it from El Reg or from the blog of the security researcher that discovered the bug – Russ McRee:

Following the principles of one flaw to rule them all, a single validation error in the q variable found in http://[Insert customer here].com/socialsearch/query?cn=[customer]&cc=us&q= led to numerous Baynote customers falling prey to cross-site scripting. [VIDEO HERE]

I don’t know if Baynote contacted their clients to explain (a) the ramifications of the flaw and (b) that they were making code changes in the background…but either way, I have a question:

Given that a Cross Site Scripting flaw can be exploited to attack the users of a website, where does that leave the visitors of the SaaS clients’ website who would be potentially exposed to the flaw?

Along with the many benefits of SaaS services, you and your customers inherit the security bugs too.  From a business perspective we can begin the chorus of wailing and gnashing of teeth as we are reminded that a single vulnerability in a multi-tenant application, exposes all the tenants.  But what about the customers of the tenants?  Surely, the end-user is the real victim!

The positive side of this particular story is that Baynote moved quickly to fix the flaw.

The other angle on this incident is the disparity between the security claims  a SaaS/Cloud provider makes and the reality.

A quick Google site search of Baynote.com for the word ’security’ brings up this:

When a providers primary website property is vulnerable to a basic XSS attack, what do you make of statements like this?

Google Docs has just notified users affected by a flaw in its document sharing feature. We don’t know how many users or documents were affected - Google have only stated that less than 0.05% of all documents are affected (via TechCrunch).

The weakness appears to have been discovered by Richard De Vries.  He retells the story on Slashdot:

I work for a small Dutch company that uses Google Apps. This means that we can share documents with users within our domain (www.deondernemers.nl), as well as @gmail.com accounts or other Apps-domains. About three weeks ago, we discovered that some fifteen documents and spreadsheets were unintentionally shared with a lot of people, some of whom were outside of our domain. We found out that one of us had been wanting to share these documents with a colleague (within our domain). He selected the documents on the documents list and added one user. Google Docs then shared all these documents with everyone who had access to one of the selected documents.

The official Google notification to affected users describes the issue as follows:

We wanted to let you know about a recent issue with your Google Docs account. We’ve identified and fixed a bug which may have caused you to share some of your documents without your knowledge. This inadvertent sharing was limited to people with whom you, or a collaborator with sharing rights, had previously shared a document. The issue only occurred if you, or a collaborator with sharing rights, selected multiple documents and presentations from the documents list and changed the sharing permissions. This issue affected documents and presentations, but not spreadsheets.

To help remedy this issue, we have used an automated process to remove collaborators and viewers from the documents that we identified as being affected. Since the impacted documents are now accessible only to you, you will need to re-share the documents manually. For your reference, we’ve listed below the documents identified as being affected…

I’ll leave others to comment on the obvious privacy issues, but what I find really interesting is how Google handled it. Based on publicly available timeline information, we can attempt to glean insights into how the investigation progressed.

Timeline

February 22nd: Initial report from Richard De Vries.  Richard noted difficulty in knowing where he could report the issue (an issue in itself) so ended up reporting it in a general catch-all bug reporting form.

February 25th: Google Security contact Richard to verify the vulnerability report.  They requested he re-create the sharing scenario.  The 3 day delay between Richared reporting the bug and the response from Google Security will not surprise anyone thats works in a large company.  If security concerns don’t follow your established communication channels (intuitive to customers or not), then a game of pass the parcel ensues until it reaches the internal security team.

March 3rd: Google Security confirms the weakness

March 7th: Google notifies affected customers

Tasks

So in the past 10 days, these are the likely tasks that happened at Google (assume some tasks carried out in parallel and some repeat tasks):

• develop a strategy to handle the weakness and convene a virtual team with representatives from development, Google Security, operations, support, privacy, product managment and PR
• establish a meeting schedule and agree how to collaborate (Google Docs anyone? ;-)
• fully understand the problem and establish the implications of the vulnerability. This is likely to involve a mix of brainstorming, source code review and ad hoc data-mining of the Google Docs database to validate hypothesis
• perform an in-depth security source code review of the affected code and neighbouring code to check for similar logic flaws.  This larger review is a precautionary measure to identify possible weaknesses that could become the subject of attack when word of the original vulnerability gets out
• create necessary fixes and test on a standby system
• go through internal change control processes to deploy and confirm necessary fixes to some portion, or all of the production systems.  When test results match expectations, deploy globally.  Its unknown how fast Google can ripple a functionality change across their numerous data centers (anyone who knows, please share in the comments)
• identify the users affected by the flaw.  They now have to develop, test and run a query against the Google Docs data repository (BigTable?) to identify which documents are affected and in turn, which users own those documents
• having reviewed the results they now run an update to ‘reset permissions’ on affected documents (they could do the above step and this step in one script but splitting into a 2 step approach carries less risk even if it takes longer) so that only the owner has access
• the Google Docs product manager  most likely consulted with the Google privacy and legal teams to understand any potential liability
• create an email template and list for notifying users and get the text of the notification email approved by the product manager, privacy, PR and possibly legal
• the PR team prepare standard responses ready to respond to comments from the media
• prime Google Support ready to handle queries from concerned users
• issue the notification email to affected users.  Note, at time of posting, Google had not posted this issue to either their security blog or the Google Docs blog.  Given their strategy to identify, notify and protect affected users (the “sniper” IR strategy ;-), its arguable they don’t actually need to although I suspect they might
• create Google alerts on phrases in the notification email to see on which web properties it is reported to perform reputation monitoring ;-)
• perform a Root Cause Analysis (RCA) on the coding snafu and implement changes to reduce/eliminate future occurences

Even if they skipped some of the non-essential steps, that’s a *lot* to achieve in the timeframe especially as we don’t know what other issues Google Security may be fielding at the same time (its rare that only person reports an a potential security weakness in widely used software/services, let alone additional security reports that range from completely bogus to potentially valid).

In a nutshell, I’m impressed with the speed Google executed on this.  To put their response in perspective, compare how long it takes some credit card payment processors to react to major data breaches.

Hats off to Google Security on this one…