Cloud Security

The Electronic Privacy Information Center (EPIC) has lodged a formal complaint about Google to the US Federal Trade Commission (FTC), insisting that they investigate the adequacy and sufficiency of Googles privacy and security safeguards.  EPIC is also seeking changes to Googles Term of Service and a suspension of Googles Cloud Computing services until ’safeguards’ are verifiably established.  Finally, they want the FTC to “compel Google to contribute 5,000,000USD to a public fund that will help support research concerning privacy enhancing technologies…”.

EPIC forwards this complaint on 3 primary fronts:

• the specific ways that Google represents their security controls to consumers (yet disclaims all responsibilities in the Terms of Service)
• the “harm” caused by the recent Google Docs privacy breach
• the claim that Google has “inadequate security”

Secondary arguments include citing a number of other, older vulnerabilities in Google online services and referencing some significant privacy breaches where the FTC acted before.  In my view, these are distractions and inconsistent with the primary argument.  The call for Google to pay 5 million dollars is poorly framed, seemingly an afterthought and potentially devisive.  I suspect EPIC will have lost the goodwill of privacy moderates by making such a demand.  Had they just dropped the number and left the call for a fund, it might have made it more palatable.

Given the complaint is 15 pages long, there is plenty to comment on.  For the sake of brevity, lets contain our analysis to the primary arguments, introduce a potential curveball and go “one step beyond” to examine the implications for Google users should the FTC rule in EPICs favour.

What Google Says About Security

EPIC highlights two specific security claims made by Google.

On the Google Docs homepage

Getting to know Google Docs> Saving your presentation

The complaint then goes on to suggest that Google “encourages users to add personal information to their documents and spreadsheets” and repeats the statement made by Google that “your data is private, unless you grant access to others and/or publish your information”.

Having built their primary argument based on public statements made on Google online properties, they bring out the Google Terms of Service which states in Section 14.1 that the services are provided “as is”, with no warranty and that Google does “not represent or warrant” that [14.2 B] “your use of the Services will be uninterrupted, timely, secure or free from error”.  Section 15 states that Google will not be liable for losses.

The Harm Caused By The Google Docs Privacy Breach

EPIC then attempts to link the Google Docs privacy breach with harm experienced by Google users:

Curious.  2 sentences in a 15 page report where EPIC could have firmly established the ‘harm’ case.  No examples, no quantification, no impact analysis. Perhaps EPIC is playing its hand carefully and is readying a parade of impacted users who can demonstrate they were “harmed” by the privacy snafu.  Failing that, it would mean they have built their case on the morality of a software privacy bug at a popular online service and ultimately, an industry wide disparity between the big print a company uses to market their services (and software!) as trustworthy and the small print where the lawyers spell out the case for the defense.

The Claim That Google has Inadequate Security

The third and final primary argument.  Skipping past the reminder from EPIC to the FTC that they acted in response to other privacy breaches, EPIC goes on to state that Google’s “inadequate security is an unfair business practice” and that Google’s “Inadequate Security” is a deceptive trade practice”.   They argue that the Google Docs privacy breach was a result of inadequate security practices and that Google:

• encourages people to share “sensitive” documentation in promoting their services
• “knew that Cloud Computing Services are susceptible to data breaches”
• “knew that disclosure of personal user data could cause substantial injury to customers”
• was “aware that commonsense security measures, including storing user data in encrypted form, rather than in clear text, could reduce the likelihood and extent of consumer injury”
• “created an unnecessary risk to users’ data by employing unreasonable security practices, including the storage and transmission of personal information on its computer network in clear text”

What I find fascinating about this is that EPIC is drawing a significant conclusion about Google’s security practices based on the fact that Google doesn’t take “commonsense” security measures. In other words, because Google hasn’t implemented a PKI and DRM for document sharing in a (for many) free service, Google is somehow employing unreasonable security practices (!).

This just strikes me as really unreasonable and wholly unrealistic.  If the FTC mandated those level of security protections to “qualify” for accepting data that consumers choose to put in the Cloud, you can say goodnight to *all* of the popular Web 2.0 services.

With Google thoroughly chastised, they draw the following “big picture” conclusion:

First Google, now the Cloud Computing as a whole - I’d better change my domain name fast! ;-).

A Potential Curveball

After I posted my thoughts on how Google Security responded to the Google Docs sharing problem I was contacted by a Google Docs user who stated that he reported a sharing problem to Google in January. He discovered a large number of documents shared with over a hundred people (he gave specific numbers that I’m intentionally not quoting to protect his privacy).  He states he called Google Tech support who initiated a support case.

Assuming this is true (and based on his note, I have no reason to doubt it) it means either:

• Google Docs has suffered another sharing problem that was quietly fixed (no notification?)
• If this is the same sharing problem, it means at least someone in Google knew about it from late January which completely changes how their responsiveness to dealing with this problem will be perceived.

What If EPIC Gets Their Way

If the FTC went as far as forcing Google to suspend its services, we will witness the largest Denial of Legitimate Service (DoLS) attack in history.

Can you imagine how that would play out?  I suspect it would also be the worst PR disaster of all time for EPIC as Google users turn on them in their droves…

In their concern for privacy, one part of security that EPIC seems to have forgotten is availability and the Cloud is all about that.

Google Docs has just notified users affected by a flaw in its document sharing feature. We don’t know how many users or documents were affected - Google have only stated that less than 0.05% of all documents are affected (via TechCrunch).

The weakness appears to have been discovered by Richard De Vries.  He retells the story on Slashdot:

I work for a small Dutch company that uses Google Apps. This means that we can share documents with users within our domain (www.deondernemers.nl), as well as @gmail.com accounts or other Apps-domains. About three weeks ago, we discovered that some fifteen documents and spreadsheets were unintentionally shared with a lot of people, some of whom were outside of our domain. We found out that one of us had been wanting to share these documents with a colleague (within our domain). He selected the documents on the documents list and added one user. Google Docs then shared all these documents with everyone who had access to one of the selected documents.

The official Google notification to affected users describes the issue as follows:

We wanted to let you know about a recent issue with your Google Docs account. We’ve identified and fixed a bug which may have caused you to share some of your documents without your knowledge. This inadvertent sharing was limited to people with whom you, or a collaborator with sharing rights, had previously shared a document. The issue only occurred if you, or a collaborator with sharing rights, selected multiple documents and presentations from the documents list and changed the sharing permissions. This issue affected documents and presentations, but not spreadsheets.

To help remedy this issue, we have used an automated process to remove collaborators and viewers from the documents that we identified as being affected. Since the impacted documents are now accessible only to you, you will need to re-share the documents manually. For your reference, we’ve listed below the documents identified as being affected…

I’ll leave others to comment on the obvious privacy issues, but what I find really interesting is how Google handled it. Based on publicly available timeline information, we can attempt to glean insights into how the investigation progressed.

Timeline

February 22nd: Initial report from Richard De Vries.  Richard noted difficulty in knowing where he could report the issue (an issue in itself) so ended up reporting it in a general catch-all bug reporting form.

February 25th: Google Security contact Richard to verify the vulnerability report.  They requested he re-create the sharing scenario.  The 3 day delay between Richared reporting the bug and the response from Google Security will not surprise anyone thats works in a large company.  If security concerns don’t follow your established communication channels (intuitive to customers or not), then a game of pass the parcel ensues until it reaches the internal security team.

March 3rd: Google Security confirms the weakness

March 7th: Google notifies affected customers

Tasks

So in the past 10 days, these are the likely tasks that happened at Google (assume some tasks carried out in parallel and some repeat tasks):

• develop a strategy to handle the weakness and convene a virtual team with representatives from development, Google Security, operations, support, privacy, product managment and PR
• establish a meeting schedule and agree how to collaborate (Google Docs anyone? ;-)
• fully understand the problem and establish the implications of the vulnerability. This is likely to involve a mix of brainstorming, source code review and ad hoc data-mining of the Google Docs database to validate hypothesis
• perform an in-depth security source code review of the affected code and neighbouring code to check for similar logic flaws.  This larger review is a precautionary measure to identify possible weaknesses that could become the subject of attack when word of the original vulnerability gets out
• create necessary fixes and test on a standby system
• go through internal change control processes to deploy and confirm necessary fixes to some portion, or all of the production systems.  When test results match expectations, deploy globally.  Its unknown how fast Google can ripple a functionality change across their numerous data centers (anyone who knows, please share in the comments)
• identify the users affected by the flaw.  They now have to develop, test and run a query against the Google Docs data repository (BigTable?) to identify which documents are affected and in turn, which users own those documents
• having reviewed the results they now run an update to ‘reset permissions’ on affected documents (they could do the above step and this step in one script but splitting into a 2 step approach carries less risk even if it takes longer) so that only the owner has access
• the Google Docs product manager  most likely consulted with the Google privacy and legal teams to understand any potential liability
• create an email template and list for notifying users and get the text of the notification email approved by the product manager, privacy, PR and possibly legal
• the PR team prepare standard responses ready to respond to comments from the media
• prime Google Support ready to handle queries from concerned users
• issue the notification email to affected users.  Note, at time of posting, Google had not posted this issue to either their security blog or the Google Docs blog.  Given their strategy to identify, notify and protect affected users (the “sniper” IR strategy ;-), its arguable they don’t actually need to although I suspect they might
• create Google alerts on phrases in the notification email to see on which web properties it is reported to perform reputation monitoring ;-)
• perform a Root Cause Analysis (RCA) on the coding snafu and implement changes to reduce/eliminate future occurences

Even if they skipped some of the non-essential steps, that’s a *lot* to achieve in the timeframe especially as we don’t know what other issues Google Security may be fielding at the same time (its rare that only person reports an a potential security weakness in widely used software/services, let alone additional security reports that range from completely bogus to potentially valid).

In a nutshell, I’m impressed with the speed Google executed on this.  To put their response in perspective, compare how long it takes some credit card payment processors to react to major data breaches.

Hats off to Google Security on this one…