Secure delete: why is more than one pass needed?

My mom's new computer came with some software to perform a secure delete, and after reading the advertising she asked me why it was necessary. The advertising brags about 7-pass and 35-pass options to make sure your data does not fall into the wrong hands. However, my mom just didn't get it, she thought that the data should be gone if you delete the file. I was able to answer the first question, explaining that delete just removes the index entry that refers to a given file. The data will still be there until it gets overwritten and with the right software it can be recovered.

I wasn't as prepared to answer the follow up question, why is more than one pass needed? Ok, so we overwrite the file once, how can it then be recovered? I didn't have a good answer, but bumbled through a guess that it was probably like a notepad where writing on the top sheet leaves traces on the pad even after the sheet is removed. To get a better idea of how people recover data, a friend pointed me to an excellent article by Peter Gutmann called Secure Deletion of Data from Magnetic and Solid-State Memory. He gives a nice summary of the basic idea:

In conventional terms, when a one is written to disk the media records a one, and when a zero is written the media records a zero. However the actual effect is closer to obtaining a 0.95 when a zero is overwritten with a one, and a 1.05 when a one is overwritten with a one. Normal disk circuitry is set up so that both these values are read as ones, but using specialised circuitry it is possible to work out what previous "layers" contained. The recovery of at least one or two layers of overwritten data isn't too hard to perform by reading the signal from the analog head electronics with a high-quality digital sampling oscilloscope, downloading the sampled waveform to a PC, and analysing it in software to recover the previously recorded signal.

Sometimes an oscilloscope may not be enough and you might need to use magnetic force microscopy or other techniques that require very expensive equipment. It should also be pointed out that the article was written 15 years ago, and hard drive densities have increased a lot in that time period. Microscopy techniques have no doubt improved as well, but it is still going to be much more difficult to recover data from modern drives. In the 2006 NIST Guidelines for Media Sanitization, they suggest that a single pass is enough to clear data:

For some media, clearing media would not suffice for purging. However, for ATA disk drives manufactured after 2001 (over 15 GB) the terms clearing and purging have converged. Studies have shown that most of today’s media can be effectively cleared and purged by one overwrite using current available sanitization technologies.

In short, it appears to be cost prohibitive to recover data that has been wiped with a single pass. Lets face it, for most of the data on your computer it would probably cost more to recover than the attacker could ever get back by stealing that information, and most likely there are much faster and easier ways to steal your data.

God's body count in perspective

I recently finished reading Drunk with Blood: God's killings in the Bible, and I was curious how God would stack up with some of the more recent mass murderers. In particular, I chose some of the names that come up frequently including Adolf Hitler, Joseph Stalin, Mao Zedong, and Pol Pot. I also included two that are more contemporary and recently in the news: Osama bin Laden and Saddam Hussein. Satan's number was too small to make the cut.

Estimating the number of people killed by these individuals is difficult and it is impossible to get a precise number that is agreed on by all historians. Instead of trying I just looked around quickly and included a low and high estimate. This approach is similar to the book that includes a count where the Bible provides actual numbers and another count that estimates the number killed when actual numbers are not provided. One difference however, the estimate from the book of God's killings is probably on the low side where the high estimate I'm using is probably higher than most would fairly assign to these individuals. For more nuanced estimates try Who was the Bloodiest Tyrant of the 20th Century? and 1900-2000: A century of genocides. So lets get on with it, here are the numbers:

God

Lets start with God, Steve Wells helpfully has a post with an overview of all God's killings in the Bible. The count where the Bible provides the number is 2,476,636. The estimated count for other killings where the Bible is vague is 24,634,205. Read the blog or the book if you want more information. It should be pointed out, this only includes killings mentioned in the Bible. Some may think God deserves credit for later killings as well, but they are not included in this tally.

Adolf Hitler

For the low count I used the estimated number of people killed in the Holocaust. There are various numbers that get mentioned, but 14 million seems like a reasonable estimate. The high estimate blames Hitler for all of the deaths associated with World War II, and the extreme seems to be around 78 million.

Joseph Stalin

According to wikipedia, Stalin's death count falls somewhere between 3 million and 60 million. Other sources place the actual number between 20 and 25 million. I used the estimates from wikipedia.

Mao Zedong

Mao Zedong killed somewhere between 10 million and 70 million people. The discrepancy is in part whether you include deaths due to famine from policies such as the Great Leap Forward. Basically are we counting democide or genocide.

Pol Pot

The high end estimate for Pol Pot was only around 2.5 million. Given his competition, I didn't bother with a low estimate.

Osama bin Laden

Osama bin Laden was included because he was recently killed and has been in the news a lot lately. If you look at killings he planned or ordered the number is probably around 3,500 (from 1900-2000: A century of genocides). Looking at the wikipedia article it estimated the deaths from the global war on terror at 80,000 to 1.2 million. For my purposes, Osama represents the deaths from the war on terror with an estimate of 1.2 million.

Saddam Hussein

The estimate for Saddam Hussein seems to be around 600,000.

Global Deaths per Year

In addition to various tyrants, I wanted to have some kind of baseline for the comparison. I chose to use the estimated number of people that died in 2010. This number is calculated using the crude death rate of 8.37 deaths per 1000 people over a 1 year period. If the estimated population size is 6.92 billion, then the estimate for the number of people to die per year is 57.9 million.

Infographic

So with those estimates, here is a quick graphic to try and put the number of deaths attributed to God into context with the others:

Comcast Live Chat

I hate talking to sales people. These days I expect that for most activities I should be able to accomplish everything via a website and having no interaction with an actual person. Unlike some people, I prefer this lack of interaction and fill with dread when some step mentions snail mail or having to call the company. Comcast has found a new annoyance, the Live Chat. I went to the Comcast website, filled out a form, and then the only option was to enter a live chat with a Comcast representative. The first part was an infuriating series of questions asking me to given them the information I had already entered on the form. This was followed by the representative trying to sell me a bunch of crap inform me of exciting deals. At the end of this chat a survey was provided to rate the experience. Unfortunately I didn't save the survey page, but to the best of my recollection the four questions were:

Was your problem solved?

• yes
• no

I had to answer yes, however, my problem could have easily been solved if they would have just processed the web form in a reasonable way. I gave them all of the information they needed on the form, there was no reason to do the live chat.

Would you use this service again?

• yes
• no

Well, for nice high speed internet in my area Comcast is really the only choice and there was no way to avoid the live chat on the website. So yes I would use it again.

How helpful was the Comcast representative?

• not helpful
• helpful

The representative was as helpful as she could be given the whole service, especially for my issue, was a complete waste. The sales pitches were annoying, but I'm sure the representatives are required to nag the customers with that garbage.

Was this service more or less work than you expected?

• less
• about what I expected
• a little more than I expected
• way more than I expected

I said way more than expected. In reality when I first saw the text on the form saying I would have to do the live chat to finish I was expecting a complete pain in the ass and waste of my time. So I suppose it was about what I expected. However, since there was no place on the survey for free text and the questions are not designed to get useful feedback, this last question seemed like the best option for ranking them poorly.

Damn Data::UUID

The Data::UUID module has the annoying behavior that it will not fail if you provide an invalid namespace when creating a v3 UUID. It will generate a UUID, but depending on the circumstances it may generate a different UUID in subsequent calls. Consider the following example:

#!/usr/bin/perl

use strict;
use warnings;

use Data::UUID;
use UUID::Tiny;

sub v3_data_uuid {
    my $namespace = shift;
    my $name = shift;
    my $ug = shift;
    $ug = new Data::UUID unless defined $ug; 
    return lc($ug->create_from_name_str($namespace, $name));
}

sub v3_uuid_tiny {
    my $namespace = shift;
    my $name = shift;
    return UUID_to_string(create_UUID(UUID_V3, $namespace, $name));
}

# Generate a v3 UUID using Data::UUID
print "Data::UUID\n";
my $ug = new Data::UUID;
print '  1. ', lc($ug->create_from_name_str(UUID_NS_DNS, 'abc')), "\n";
print '  2. ', lc($ug->create_from_name_str(UUID_NS_DNS, 'abc')), "\n";
print '  3. ', v3_data_uuid(UUID_NS_DNS, 'abc'), "\n";
print '  4. ', v3_data_uuid(UUID_NS_DNS, 'abc'), "\n";
print '  5. ', v3_data_uuid(UUID_NS_DNS, 'abc', $ug), "\n";

# Generate a v3 UUID using UUID::Tiny
print "UUID::Tiny\n";
print '  1. ', UUID_to_string(create_UUID(UUID_V3, UUID_NS_DNS, 'abc')), "\n";
print '  2. ', UUID_to_string(create_UUID(UUID_V3, UUID_NS_DNS, 'abc')), "\n";
print '  3. ', v3_uuid_tiny(UUID_NS_DNS, 'abc'), "\n";
print '  4. ', v3_uuid_tiny(UUID_NS_DNS, 'abc'), "\n";

# Generate a v3 UUID using Data::UUID with an invalid namespace
print "Data::UUID - bad namespace\n";
my $namespace = 'namespace';
print '  1. ', lc($ug->create_from_name_str($namespace, 'abc')), "\n";
print '  2. ', lc($ug->create_from_name_str($namespace, 'abc')), "\n";
print '  3. ', v3_data_uuid($namespace, 'abc'), "\n";
print '  4. ', v3_data_uuid($namespace, 'abc'), "\n";
print '  5. ', v3_data_uuid($namespace, 'abc', $ug), "\n";

# Generate a v3 UUID using UUID::Tiny with an invalid namespace
print "UUID::Tiny - bad namespace\n";
print '  1. ', UUID_to_string(create_UUID(UUID_V3, $namespace, 'abc')), "\n";

This example also uses UUID::Tiny as a point of comparison. The output that I get when running this example is:

$ ./uuid.pl 
Data::UUID
  1. 1fc38bb9-aae5-3dba-9894-38925088c9c0
  2. 1fc38bb9-aae5-3dba-9894-38925088c9c0
  3. 1fc38bb9-aae5-3dba-9894-38925088c9c0
  4. 1fc38bb9-aae5-3dba-9894-38925088c9c0
  5. 1fc38bb9-aae5-3dba-9894-38925088c9c0
UUID::Tiny
  1. 5bd670ce-29c8-3369-a8a1-10ce44c7259e
  2. 5bd670ce-29c8-3369-a8a1-10ce44c7259e
  3. 5bd670ce-29c8-3369-a8a1-10ce44c7259e
  4. 5bd670ce-29c8-3369-a8a1-10ce44c7259e
Data::UUID - bad namespace
  1. 15da4c6e-29ae-3f6c-a7ed-ec36373d4e5d
  2. 15da4c6e-29ae-3f6c-a7ed-ec36373d4e5d
  3. 8fa130be-2517-3e29-a087-c7d75545a62c
  4. 8fa130be-2517-3e29-a087-c7d75545a62c
  5. 8fa130be-2517-3e29-a087-c7d75545a62c
UUID::Tiny - bad namespace
UUID::Tiny::string_to_uuid(): 'namespace' is no UUID string! at ./uuid.pl line 50

When a valid UUID is used for the namespace, Data::UUID consistently generates the same value. However, if I pass in a bad value such as the string literal namespace, then I get different values depending on how/where/when it is called. UUID::Tiny on the other hand dies with a nice error message telling me exactly what is wrong.

Polls and Research on Public Acceptance of Evolution

It was brought to my attention that my post on the acceptance of evolution by state was cited in an ASA article Polls and Research on Public Acceptance of Evolution. The article provides a summary of around 50 sites discussing the public acceptance of evolution. The ASA, American Scientific Affiliation, is an organization of Christian scientists so I would guess they have some bias towards creationism. That said, this is the first time I have heard of the ASA and their article seems to provide a fair summary of the findings presented.

What is Life?

ASU Origins Project ASU Gammage Auditorium Feb. 12, 2011

Richard Dawkins, J. Craig Venter, Nobel laureates Sidney Altman and Leland Hartwell, Chris McKay, Paul Davies, Lawrence Krauss, and The Science Network’s Roger Bingham discuss the origins of life, the possibility of finding life elsewhere, and the latest development in synthetic biology.

SIGMOD Experimental Repeatability Requirements

I was recently pleased to hear about the SIGMOD Experimental Repeatability Requirements. The stated goal is to:

The goal of the repeatability/workability effort is to ensure that SIGMOD papers stand as reliable, referenceable works for future research. The premise is that experimental papers will be most useful when their results have been tested and generalized by objective third parties.

Apparently it was first done for SIGMOD 2008 and has been getting refined since then. This is something that I think has been lacking in computer science research for quite a while. The obvious benefit is that it makes it much harder to fake results or to report something in a paper that is not really what was implemented and tested. The other benefit is to researchers trying to build on or provide a better alternative to the method presented in a given paper. Anyone who has gone through a graduate computer science program will know the pain of trying to figure out exactly what was done in some paper using only the written description. If you are lucky there is clear pseudocode that can be translated into a working program. But then you still have to worry about things such as magic numbers used to tune the system, or whether the differences you are seeing in the results could be due to other factors such as the machine architecture. Being able to grab the code used for the original paper provides a much faster and more accurate basis for comparing it to a new method.

Unfortunately, though it seemed to be a promising idea, I think their implementation is a crock. The first problem is that participation in the repeatability program is optional:

The repeatability & workability process tests that the experiments published in SIGMOD 2011 can be reproduced (repeatability) and possibly extended by modifying some aspects of the experiment design (workability). Authors participate on a voluntary basis, but authors benefit as well:

• Mention on the repeatability website
• The ability to run their software on other sites
• Often, far better documentation for new members of research teams

The second problem is that it does not mean the code will be made available to everyone. I didn't see any mention of archiving on the 2011 description or how I would be able to get the code for a given paper. The SIGMOD 2010 requirements say the following about code archiving:

Participating in the repeatability/workability evaluation does not imply that the code will be archived for everyone to use subsequently. Pending authors' agreement, the code could be uploaded in the SIGMOD PubZone.

If I understand correctly, it means that if an author chooses to participate, then a committee for the conference will attempt to reproduce the experiments for that paper. After that is done, then the code will only be archived and made available if the author chooses. What the hell were they thinking? I would much rather get rid of the committee that tries to reproduce the results and make it a requirement that the full source code and data sets be made available as supplemental material for all papers. The code needs to be out in the open so it can be scrutinized along with the paper by the broader community of researchers. Furthermore, this supplemental material should be available to those doing the initial reviews of the paper to decide whether or not to accept it.