/ Digital Ocean

What to make of Digital Ocean's data scrubbing

It's always great to get your thoughts and decisions questioned. Yesterday, after sharing a meal and talking all things tech with a friend I was asked:

My first thought was "If you're that concerned about your data, DON'T PUT IT IN THE CLOUD!", however that's a regression to the "Security says NO" mentality that many info-sec practitioners, myself included, are trying bury deep in the past. So what do I make of it? A few things come to mind:

1) SSDs great for performance, were bad for forensics####

I haven't looked into enterprise storage SSDs, but last time I checked consumer SSDs it was really hard to verify if data on SSDs had actually been erased because of the way their firmware intercepted commands to specific blocks. This is primarily due to the way that SSDs try and level the the reads and writes across the entire "drive". This may well have changed in the last few years, or more likely be different when you're specifically zeroing the whole drive. This wasn't something I'd considered when selecting a VPS with SSD storage. I'll need to look into this again in the near future.

2) Multi-tenancy, the good, the bad, and the ugly####

Risks associated with multi-tenancy is one of the primary concerns raised when dealing with "The Cloud". One common concern is that someone within the hosting provider could access your data. Per yet another Securosis posting1 it should generally be assumed that your hosting provider can access all of your data (that's not to say they are) unless you're encrypting your data with keys that you control.

This angle of Digital Ocean not properly scrubbing their data expands the concern from just the provider to other customers accessing your data. I tend to feel that this type of access would be largely opportunistic, rather than targeted. That's not to say that the damage would be lessened, but the likelihood of someone targeting you and waiting for a VM to be incorrectly destroyed are minimal.

So what could we do? The cliché rule that states "the first rule of management / being a tech leader / security / [insert profession, pretty much anything except fight club, is that it's always your problem" holds true. If you've got sensitive data on a system, you should take responsibility for protecting it and then destroying it correctly. Whilst Digital Ocean did change the rules when they updated their process to not scrub and not informing anyone, relying on a third party to destroy your own data doesn't make sense to me. Especially when it's being hosted on someone else's infrastructure.

3) Highlights what we take for granted when IT is done internally####

For those of us who have worked as System Administrators at some point it can bring back memories of working out the best and quickest way to securely erase systems prior to disposal. Let alone the conundrum of how you should handle broken drives that are to be sent back to your storage provider under warranty.

I'd hope that most mature organisations have an information management life-cycle, with the final stage being destruction. When we start moving systems out to the cloud, and rely on developers 2 not versed in these concepts, we open ourselves up to exposure. It brings to mind the idea of a security poverty line as described by Wendy Nather from 451 Research3, which is going to become more and more significant as people move their infrastructure, platforms, and software to the cloud.

There is some good work being done by companies like Casey Ellis' Bugcrowd, who are hoping to lower the level of entry for Bug Bounties. However there is a need to simplify what happens in the cloud for those not dealing with Security on a day-to-day basis.

4) This would be an interesting research topic####

I don't recall hearing of any research across the different hosting providers to see if any data could be retrieved from previous users of shared storage. Some quick searches of the internet, and reading the Digital Ocean blog post's comments 4 show that many other hosting providers have battled with this. Some have ensured their processes do scrub all data, or some specifically state in their T&Cs that there is no guarantee that scrubbing occurs.

This could be a gold-mine of data, but more than likely just a bunch of very similar cloud images that don't have much sensitive data.

5) Encryption to the rescue...?####

Maybe encryption is the solution to this. Last time I heard, encrypting the System / Boot drive on a cloud hosted system isn't the easiest. Maybe encrypting the data or data drives is an alternative approach. Either solution could have reduced the impact of this mistake.

If we want encryption to be used it has to be simple and easy enough for anyone spinning up their own machines. If your organisation has the luxury of a dedicated security or server admin team this might be achievable. However I'm assuming that most VPS users are small organisations quickly trying to get something up and running with the lowest entry cost.

Why didn't I focus on Digital Ocean?####

I haven't spent too much time thinking about Digital Ocean's specific problem. I wasn't aware of the problem when I first selected them. Would it have made a difference in me selecting them? Probably not.

For me, my machine on Digital Ocean is a sandbox or playground. I'm not storing anything sensitive, nor am I reliant on Digital Ocean to run anything mission critical. And as you can see from my last post I selected Digital Ocean because of pricing and am trying to understand any security concerns by simply running this site as most other people would. Does it mean that others should take notice? By all means, but I think these same concerns should be across any shared hosting provider.

All of that said, I recently migrated my droplet (VM) from New York to San Francisco and did double check that the "Scrub Data" option was selected by default when I went to destroy my old droplet.

I think that Digital Ocean got their initial response to the github concerns wrong, had to eat some humble pie, and then had to quickly update their processes. Their full response can be found in the footnotes 4, along with many unhappy customer responses.

In this day and age of immediate communications companies responses will, at times, miss the mark. When security is at stake this could, ever so slightly, be to their detriment. If we look back at the large data-breaches or security mess-ups over the last few years I can't think of many companies that have had any long term adverse reactions (with the exception of distribute.IT). Digital Ocean have also just reached over 1 million VMs and don't appear to be stopping any time soon.

Most commonly these mess-ups reverberate around the InfoSec Echo-Chamber, and then end up as an illustration of how not-to-do-things in a conference slide deck. Or worse still, these mess-ups end up in some marketing material for an InfoSec company - which I'm sure we'd all dread!


1 - https://securosis.com/blog/how-to-tell-if-your-cloud-provider-can-read-your-data-hint-they-can

2 - I'm not having a go at developers, they have skills that I'll never have. Digital Ocean's target audience is developers as evidenced by their tagline of *"Built for Developers"*

3 - https://451research.com/t1r-insight-living-below-the-security-poverty-line

4 - https://www.digitalocean.com/blog_posts/transparency-regarding-data-security