Yes, you do need stats ... actually

Yesterday, I received the good news that my validation study of how a course in statistics could improve the statistical literacy of digital / multimedia forensic analysts when delivered on-line as micro-learning was published by the Chartered Society of Forensic Science in the UK. I got excited and put the good news on my LinkedIn feed.

Along with the usual emoji responses, I received the comment shown below.

Rather than simply comment there, I'd like to take the opportunity to illustrate the many ways in which it's not just me who says that the world of the digital evidence analyst can benefit from a solid foundation in statistics.

You see, the course was created because the relevant government bodies around the world have said, on a rather regular basis, that the investigative services and the forensic sciences need a solid foundation in statistics.

Starting at the US government level, there's the PCAST Report from 2016 (link): "NIST has also taken steps to address this issue by creating a new Forensic Science Center of Excellence, called the Center for Statistics and Applications in Forensic Evidence (CSAFE), that will focus its research efforts on improving the statistical foundation for latent prints, ballistics, tiremarks, handwriting, bloodstain patterns, toolmarks, pattern evidence analyses, and for computer and information systems, mobile devices, network traffic, social media, and GPS digital evidence analyses." (emphasis is mine)

CSAFE has already responded with some tools for digital forensic analysts (link).  The ASSOCR tool will help analysts "determine if two temporal event streams are from the same source by through this R package that implements a score-based likelihood ratio and coincidental match probability methods."

The HEISENBRGR toolset can be used to "match accounts on anonymous marketplaces, to figure out which of them belong to the same sellers."

What about NIST? What is the issue that NIST is taking steps to address? The PCAST report notes, "The 2009 NRC report called for studies to test whether various forensic methods are foundationally valid, including performing empirical tests of the accuracy of the results. It also called for the creation of a new, independent Federal agency to provide needed oversight of the forensic science system; standardization of terminology used in reporting and testifying about the results of forensic sciences; the removal of public forensic laboratories from the administrative control of law enforcement agencies; implementation of mandatory certification requirements for practitioners and mandatory accreditation programs for laboratories; research on human observer bias and sources of human error in forensic examinations; the development of tools for advancing measurement, validation, reliability, and proficiency testing in forensic science; and the strengthening and development of graduate and continuous education and training programs."

It's that last bit that prompted me to design and validate an instructional program in statistics for forensic analysts. But, it's the first sentence that speaks to the comment from LinkedIn. Analysts don't deal in absolutes or definite - binary. The world of the computer program may be binary, but the world certainly isn't. There is a natural variability to be found everywhere. But more to the comment's point, how does an analyst know that their "various forensic methods are foundationally valid, including performing empirical tests of the accuracy of the results."

Ahh... but, you're saying, all of your support is from the United States. It doesn't apply to the rest of the world. In that, you're wrong. Let's look at the UK.

In September 2018, Members of the Royal Statistical Society Statistics & Law section (link) submitted evidence (link) to a House of Lords Science and Technology Committee inquiry on Forensic Science. Question 2 asked, "what are the current strengths and weaknesses of forensic science in support of justice?" Here's the RSS' response. Notice the imbalance between strengths and weaknesses.

I've highlighted the relevant section as it relates to this topic. "poor quality of probabilistic reasoning and statistical evidence, for example, providing irrelevant information because the correct question is not asked. For example, an expert focused on the rarity of an event, rather than considering two competing explanations of an event."

Our course on statistics for forensic analysts seeks to teach probabilistic reasoning, exploring the differences between objective and subjective statistics, as well as the fact that most of the forensic sciences currently work in the wold of abductive reasoning (taking your best shot).

Now there's the accusation that digital analysts are often engaged in "push button forensics." We buy tools from vendors and hope that they're fit for purpose and accurate in their results. But are they? We don't know, so we validate our tools (hopefully). If you're trusting the market to deliver reliable, valid, and accurate tools, you may be disappointed. As the above referenced report notes, "What can be learned from the use of forensic science overseas? Seen from continental Europe, there has been a loss of an established institution (FSS) with a profound body of knowledge. Now research seems scattered among different actors (mainly academic), as commercial providers might have other priorities and limited resources to invest in fundamental research." (emphasis mine)

To the Royal Society's point, if you're a digital analyst and there's a challenge to your conclusions or opinions, on what do you base your response or your work? For example, you've retrieved photos from a computer or phone. Your tool automatically hashes the files. But, a cryptographic hash does not guarantee the authenticity of the file, only places a unique value into the process to handle questions of integrity. How do you conduct an authenticity examination without a knowledge of statistics? You can't. How do you validate your tools without a knowledge of statistics? You can't.

Over in Australia (link), there is agreement on the need for training and research - just what I've presented. "There is however one aspect of the report with which the Society is in complete agreement; the need for both continuous training and research in forensic science. We are also aware of the lack of funding for this research and therefore support the recommendation of PCAST that this is essential if our science is to continue to develop into the future."

To conclude, yes, you do need training / education in statistics if you're engaged in any forensic science discipline. Many practitioners arrive in their fields with advanced college degrees and thus will have had exposure to stats in college. But, on the digital / multimedia side, many arrive in their fields from the ranks of the visible policing services. They may not have a college degree. They may only have tool-specific training and may be completely unaware of the many issues surrounding their discipline. It's for this group that I've designed, created, and now validated my stats class. It's made in the US, to be sure, but it's informed by the global resources listed in this post - and many others.

I hope to see you in class.