Validation?

We were having a discussion on the show floor at HTCIA recently. HTCIA is all about computer forensics and the things that are on the periphery of that discipline ... like cell phone forensics.

The topic of validation came up in relation to the various cell phone forensic tools that were being sold that week. How does one validate the cell phone forensic tools? How does one even choose which tools to buy? Can you believe the marketing hype from the vendors? The answer, as always ... it depends.

One of the people present gave the following example. They received a particular handset for examination in a criminal case. The court requested all text/sms messages be retrieved and made available for the court. The examiner took the phone off the network and browsed to the messages. The phone's display noted that there were 270 messages in the phone. The examiner had Secure View from DataPilot. Secure View is relatively inexpensive. It was the only tool that this examiner had as he was new to forensics and his small agency didn't have a lot of money. Problem? The handset wasn't supported by Secure View. Take a look at their site. More often than not, they don't support the retrieval of sms messages.

The problem is compounded by the fact that there isn't a single vendor that gets all phones. Most products are expensive in terms of buying the hardware and getting training on how to use it correctly. Cellebrite's UFED Physical Pro can run almost $10k. The Standard version is cheaper, and many people opt for it out of budgetary concerns. Cellebrite says that it gets so many thousand handsets, but do they really? They count in their list of supported handsets ones in which they only retrieve the phone book and call history.

If you are ordered by the court to retrieve all sms messages, what do you return? Just those visible? Visible and deleted? How do you interpret all sms messages? What if your tool doesn't support the handset? What if your tool doesn't support the retrieval of deleted files from that handset? What do you do? Are you ethically bound to recuse yourself do to the lack of appropriate tools?

Can you consider your tool valid if it only retrieves data from certain handsets? Is it therefore valid if it retrieves some things, but not everything? As an example, Secure View does not support the retrieval of sms messages from any HTC phone. It'll get other things from HTC phones, but not sms messages. Is it a valid tool for use with HTC phones?

Enter Crawford and Brady. You get an order to retrieve sms messages from a handset. Your tools don't support that particular handset. Does your attorney goes to trial not knowing what's in those messages? The other side employs an expert with tools that do support the retrieval of all sms messages from the handset. Are you and your attorney comfortable with this scenario? What about a scenario where your tool gets most, but not all messages. Your visual inspection shows 270 messages. Your tool retrieves 249 messages. Where'd the other messages go? Try as you might, you can't get to the other messages in your retrieval. What do you do? Is your process valid?

Crawford says that you get to be questioned. Brady says that the answers to those questions get to follow you around for the rest of your career. Given this, are you comfortable with your tools and your procedures? If you are not, do you have the institutional support to make necessary changes? What happens to you if you don't/can't? Will expediency rule, or will science win out? The answers from the HTCIA show floor might surprise you.