To save people from clicking, I'm pasting here the original thread from the (now dead) linkedin group:
Andrew said:Incident response in the cloud
I know this is something that has been touched on in other discussions but is something that I wanted to pull out as an individual concern. How does moving services to the 'cloud' impact incident response procedures and capability?
My primary concern is that if I have a sufficiently large incident to warrant it I can't go to my machine room and physically pull the plug, attach files to a write-blocker and work off a copy of the system to see what went wrong. I know a few providers allow some imaging capabilities, but does this guarantee the preservation of deleted files and other forensic artifacts? Would it stand up as evidence in court? What happens if the machine seamlessly migrated between the cloud providers hardware/datacentres after or during the incident you're trying to respond to?
I'm sure I'm not the first to have considered this, but I'm yet to find a satisfactory answer, standard response I've been given is "doesn't' matter, kill one instance and start a new, clean image" keeps you running, doesn't fix the problem. I'm hoping one of you guys can help point me to a better option.
John said:The write blocker can be virtual, along with the rest of the system. Simply configure the VM host to make the virtual drive read-only. Additionally, you can accomplish this by taking a snapshot of the virtual drive - this gives you all the benefits of a write blocker - you can see what the drive looked like at that specific point in time - plus you or the malaligns can continue to write to the drive, and you can see what changed between the snapshot and time deltas since then. Either way, you've got an exact image of the system's drive (s) at time of investigation, which I believe will qualify as a "bit-wise copy" of the system.
There's other benefits to performing forensics in a VM situation, as you can examine the state of the compromised machine from "outside" the machine - look at memory contents and what the cpu is doing.
Actually, I'd think from an operational point of view a virtualized system provides another forensic benefit - you can bring up a new system based off the "clean" image and take the compromised system off-net but leave it running and do a live investigation without worrying about the provided service being down.
Note - all of this goes out the window if the attacker manages to penetrate the guest vm and get to the host. At that point, there's all sorts of ugly going on and the VM system goes from being better for forensics to way worse, compared to a non-virtualized system.
The legal/court aspect is a good question, I haven't researched that for cloud yet, not sure if the technology is certified for use as evidence. Would love to hear feedback from others...
Craig said:Great question.
From experience, my take is that as long as you can substantiate chain of custody, you can do live forensics, offline forensics, whatever (within reason).
Again, we need to think differently depending on the cloud layer we are using and who has management responsibility.
For SaaS, forensics boils down to log analysis...
For PaaS, forensics boils down to datastore/database queries and log analysis
For IaaS, forensics is normal from layer 2 up ;-). Clearly with IaaS there is a lot more scope for traditional forensics (acquisition is obviously different though). If its your own private cloud, its down to whatever policy you have senior managements' agreement on (i.e. blanket approval that if mission critical VMs get hacked, you can take down the host itself...).
For public IaaS, as John indicates, its gonna be 100% virtual. I haven't seen any IaaS provider state that they offer IR services beyond the usual high level "buy premium support and we'll help you" fluff. If I had time, I would offer a cloud based forensics service - definite gap in the market (if there is a market for it yet). If anyone wants to talk about doing that, ping me
.
You're right to be concerned about machine migrations as it is completely possible. That could happen before you get to take the image or even during. You won't know about it.. I suspect its gonna boil down to wheeling in the cloud providers security guru as an expert witness if the other side is smart enough to even pick up on this.
Personally, I think cloud is great for forensics. It removes the physical friction involved in acquisition, you have "unlimited" storage on a pay-as-you-basis and can spin up multiple VMs to do parallel image data mining. Very cool.
I don't think our processes are going to be affected as long as we stay committed to documenting what we are doing, noting anomalies, and ensuring we are provably competent with the tech before we have an incident.
Probably doesn't answer your whole question but does it get you part way?
Tomasz said:I'm not a lawyer nor expert in cloud computing or forensics, but here are my impressions or rather questions/doubts I have. One thing is certain - I'd like to have a bit more time for experiments with IR/forensics in the 'clouds'.
Craig, I really like the concept of using multiple instances for data mining - kind of essence of cloud computing. That opens a lot of new cool possibilities and potentially not so cool liabilities. It may be very interesting experiment in fact!
I would say migrations can be a big issue not even related to forensics. If they happen across country borders (which is possible) there may cause legal issues as there are some limitations on moving certain types of data for example from EU to the US (data protection and privacy laws, etc).
Personally I would be more concerned about possibility of instance being destroyed or replaced with a clean one before you can gather any evidence - this agility is the core spirit of cloud computing, but at the same time I see it as a kind of a race condition with scenarios like auto-scaling features killing unused instance (of course it has to be the one we will want to investigate - see Murphy's Law).
John has some very good points about benefits of virtual environment in IR/forensics. I think if we take forensic limitations that come out of the cloud environment specifics, slightly modify the acquisition procedure (if needed for the case), document the process to acquire evidence, then it can be explained in the court in a logical way and I don't see a reason why the evidence couldn't stand. Will it stand - that's to be tested, but assuming that a logical approach was used to gather it in the first place, I can only hope it wouldn't be much of problem.
Any thoughts?
Craig said:Tomasz - I agree with everything you've written - all solid points. The generic data privacy/cross border data transfer issues of hypervisor mobility (the fancy term
is definitely recognised...but beyond that awareness, I haven't heard more. The key in my mind is not to store this kind of data on the compute instances. Instead, use cloud storage with meta data containing 'data mobility' restrictions and mount the volume across (when I say mount, I don't mean NFS/CIFS...more like EBS from Amazon). Again though, the capability to do this "intelligent", policy based access isn't common yet
Andrew said:Thanks for the input guys, believe you've confirmed my thinking that cloud computing will have it's advantages over traditional models from a practical standpoint, but as usual the law and procedures will spend a while playing catch-up and likely drop the ball for a while in the meantime.
On the issue of data migration, think this just comes down to points raised in other threads, you really need to read the contract and SLA to understand exactly what you are getting from your provider. If your data can't (or you don't want it to) leave a particular geographical location, pick a provider that stays in that are, or provides the ability to define acceptable locations.
Likewise if data migrations could cause a problem with data acquisition and related evidence gathering choose a provider that provides you greater control of what is migrated and when (not sure if this is current;y possible?).
Craig said: Agreed.
The bigger issues are incident response/forensics of the cloud platform itself. Obviously as a customer you have no control over this. Hence in your due diligence of cloud providers it becomes vital to establish what demonstrable capability they have.
Tomasz said:Yes Craig, the key is to keep data on the proper data store and not the compute instance, but if I was the bad guy I would use an instance to siphon out the data from the data store, doing my best to not leave a trace at the data store itself.
If I could only contaminate an instance that is quite likely to be destroyed as soon as it's not needed, then you loose any chance of doing forensics on how I got in, what were my interactions with the system, what software I used to siphon the data. Cloud environments are much more dynamic than VPSes or any other kind of VMs. This dynamics can actually help attackers :-(
Normally you do IR and then move into forensics if needed. I'm thinking that for cloud environments part of the IR should be linked with the initial part of forensics - data acquisition - just by pure assumption that the lifetime of a compute instance may be too short to do anything else. I'm talking about loosing the only opportunity to get a snapshot of what happened at the instance that had access to the data.
I'm short on caffeine so the above text is a bit chaotic, but I hope you see my point.
