NIST Forensic Science Standards Inventory Now Available Online

This just in from NIST: "NIST’s Organization of Scientific Area Committees (OSAC) is taking the first steps toward developing an OSAC Registry of Approved Standards and an OSAC Registry of Approved Guidelines.

Independent scientific working groups, standards development organizations, professional organizations and government entities have developed many standards and guidelines for use by the forensic science community. In order to capture and build upon this work, NIST Forensic Science Program staff members have compiled an inventory of these existing documents. The inventory is available to download from the OSAC Catalog of Standards and Guidelines web page as a sortable Excel spreadsheet file. It contains the titles and source information for 730 standards, guidelines and related documents. The inventory also lists web addresses for documents that are available online.

This inventory is intended to serve as a resource to the forensic science community and a foundation for the future work of the Organization and Scientific Area Committees. The current version is the result of a scan of all known forensic science organizations, associations and standards development organizations, in addition to the results of a data call to each of the 21 independent forensic science scientific working groups. The catalog contains all applicable forensic science standards, guidelines, best practices, protocols and policies.

OSAC subcommittees will review each document's relevance and validity to its forensic science discipline. The subcommittees will make recommendations on which of the existing documents should be adopted, in part or whole, by OSAC. They will also identify gaps for which new standards and guidelines should be developed.

Subcommittee recommendations will be discussed during the first public meetings of OSAC's five Scientific Area Committees, to be held Feb. 16 and 17, 2015, during the annual meeting of the American Academy of Forensic Sciences, in Orlando, Fla.
Go to the OSAC Catalog of Standards and Guidelines web page for more information and to download the spreadsheet file.

What is forensic science

As I prepare to head out to Orlando for next week's OSAC in-person meeting, I want to revisit one of the papers that the OSAC has issued since it's founding.

Consider that the OSAC is a group that includes all forensic science disciplines. Thus, in harmonizing the language used to describe what should be a simple term - forensic science - much work was done to arrive at a definition that works for all forensic science disciplines.

I've shared the highlights in several posts and papers. Here's the full discussion.

---
2. Forensic Science
A definition of forensic science should focus on the evidence scrutinized and the questions answered by the inquiry. After extensive research, surveys, and discussions, the TG formed the following understanding of the aim and purpose of forensic science:

Traces are the fundamental objects of study in forensic science. A trace is a vestige, left from a past event or activity, criminal or not. The principle that every contact leaves a trace was initially attributed to Edmond Locard, and has evolved into a new definition of the trace to include a lacuna in available evidence, as well as activities in virtual settings (Jaquet-Chiffelle, 2013):
A trace is any modification, subsequently observable, resulting from an event.

This is not to suggest that all forensic questions involve event reconstruction, merely that all traces involve some modification. Even immutable objects can be a trace when their occurrence in relation to a forensic inquiry is the consequence of an event (e.g., a mobile device identifier deposited at a crime scene, or DNA transferred onto a victim). The modification can affect an entity in an environment or the environment itself. Its nature can be physical or virtual, material or immaterial, analog or digital. It can reveal itself as a presence or as an absence.

Forensic science addresses questions, potentially across all forensic disciplines. These questions are addressed using a specific and finite number of core forensic processes. For the purpose of this document, these processes are labeled as: 1) authentication, 2) identification,
3) classification, 4) reconstruction, and 5) evaluation.

The following definition of forensic science emerged from this work:
The systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context.

The term systematic in this definition encompasses empirically supported research, controlled experiments, and repeatable procedures applied to traces. The term coherent entails logical reasoning and methodology. This definition uses legal context in the broadest terms, including the typical criminal, civil, and regulatory functions of the legal system, as well as its extensions such as human rights, employment, natural disasters, security matters.

---

Continuing on ...

---

3. Digital/Multimedia Evidence
To understand the scientific foundations of digital/multimedia evidence and how this fits into forensic science, it is necessary to consider the specializations of digital/multimedia evidence. Digital/multimedia evidence encompasses the following sub-disciplines (ed. note: edited for brevity), which are organized according to the current OSAC structure:

Video/image technology and analysis: handling images and videos for forensic purposes. This includes classification and identification of items, such as comparing an item in an image or video with a known item (e.g., car, jacket). This also includes authentication of images and videos, metadata analysis, Photo Response Non-Uniformity (PRNU) analysis, image quality assessment, and detection of manipulation. Operational techniques include image and video enhancement and restoration.

Digital evidence: handling digital traces for forensic purposes, including classification and identification of items, activity reconstruction, detection of manipulation (e.g., authentication of digital document, concealment of evidence). Within the current OSAC structure, audio recordings are treated as a form of digital evidence for enhancement and authentication purposes.

The foundational sciences for the various sub-disciplines of digital/multimedia evidence are primarily biology, physics, and mathematics, but also include: computer science, computer engineering, image science, video and television engineering, acoustics, linguistics, anthropology, statistics, and data science. Principles of these, and other disciplines, are applied to the traces, data, and systems examined by forensic scientists. Study of foundational principles in digital/multimedia evidence is ongoing, with consideration for their suitability in forensic science applications.

Furthermore, many digital traces are changes to the state of a computer system resulting from user actions. In this context, the discovery of principles in how computer systems function, is a fundamental scientific aspect of digital/multimedia evidence. The systematic and coherent study of digital/multimedia evidence is made more complicated by the evolving nature of technology and its use. While the foundations of digital/multimedia evidence are largely in computer science, computer engineering, image science, video and television engineering, and data science, the underlying digital traces are, in large part, created by actions of operating systems, programs, and hardware that are under constant development. As a result, it will not always be possible to test in advance the performance of such systems under every possible combination of variables that may arise in casework. However, it may be possible, to test the performance of a particular system under a particular set of variables in order to address questions arising in a specific case. For instance, digital documents created using a new version of word processing software can exhibit digital traces that were not previously known. The observed traces can be understood by conducting experiments; studying the software under controlled conditions. In this manner, generalized knowledge of digital/multimedia evidence is established and can be used by any forensic scientists to obtain reproducible, widely accepted results.

---

It's this last paragraph that I'll finish with. Notice these statements:

• "However, it may be possible, to test the performance of a particular system under a particular set of variables in order to address questions arising in a specific case."
• "The observed traces can be understood by conducting experiments; studying the software under controlled conditions. In this manner, generalized knowledge of digital/multimedia evidence is established and can be used by any forensic scientists to obtain reproducible, widely accepted results."

These statements have to do with validation and experimental design. Are you validating your tools? Are you conducting experiments, following the rules of experimental design?

If you'd like to explore these concepts, we've got classes that address most of the topics illustrated in this section of the document. Check out our calendar. If you find a date / class that works for your schedule, sign up. If you can't find a date that works, suggest one. We're here to help.

See you in Orlando.

NIST Webcast on New Forensic Scientific Area Committees Set for Feb. 18, 2014

February 14, 2014

Contact: Linda Joy
301-975-2757

The NIST forensics science team is holding a public presentation and live webcast on its new Organization of Scientific Area Committees (OSAC) at the American Academy of Forensic Sciences annual meeting in Seattle, Feb. 18, 2014, 5:00 to 7:00 p.m. PST.

The establishment of OSAC is a result of a NIST and Department of Justice memorandum of understanding signed last year with the aim of improving standards and practices in forensic science laboratories.

NIST envisions uniform administration of development, promulgation and adoption of standards through the OSAC as well as supporting communication flow between the scientific area committees and the forensic science community. The intent is to bring structure, scientific rigor and increased communication among forensic scientists, research scientists, academicians, statisticians, attorneys, managers and quality assurance specialists.

The presentation will take place at the Washington State Convention Center, Ballroom 6C, Seattle, Wash.

Agenda
5:00 to 5:10: Introduction – Mark Stolorow
5:10 to 5:15: Background – Rich Cavanagh
5:15 to 5:30: Notice of Inquiry Responses – Susan Ballou
5:30 to 6:00: OSAC Plan – Barbara Guttman and John Butler
6:00 to 6:15: OSAC Membership – John Paul Jones II
6:15 to 7:00: Questions

Webcast Link: http://psav.mediasite.com/mediasite/Play/8e65ad3f9958440b8eab9da0831e3b1d1d
The presentation slides will be posted here on Tuesday, Feb. 18.

Important notes about Live Webcast:
This webcast will be streamed from Mediasite by sonicfoundry, individuals logging in from personal computers/laptops may need to:
Install Microsoft Silverlight prior to logging on – this is needed in order to view the program.
If any participants use Safari on Mavericks, they will need to disable plugins power saver option. That should be turned off so that it does not prevent the media player from loading.

Report writing in forensic multimedia analysis

You've analyzed evidence. You've made a few notes along the way. You've turned those notes over to the process. Your agency doesn't have a specific requirement about what should be in your notes or your report or how detailed they should be. In all the cases that you've worked, you've never been asked for specifics / details.

Now, your case has gone to trial. An attorney is seeking to qualify you to provide expert (opinion) testimony. They introduce you, your qualifications, and what you've been asked to do. The judge may or may not declare you to be an expert so that your opinion can be heard.

As a brief aside, your title or job description can vary widely. I've been an analyst, specialist, director, etc. FRE Rule 702, and the similar rule in your state's evidence code, governs  your testimonial experience. Here's the bottom line: according to evidence code, you're not an "expert" unless the Judge says so, and then only for the duration of your testimony in that case. After you're dismissed, you go back to being an analyst, specialist, etc. You may have specific expertise, and that's great. But the assignment of the title of "expert" as relates to this work is generally done by the judge in a specific case, related to the type of testimony that will be offered.

A technician generally offers testimony about a procedure and the results of the procedure. No opinion is given. "I pushed the button and the DVR produced these files."

An expert generally offers opinion based testimony about the results of an experiment or test. "I've conducted a measurement experiment and in my opinion, the unknown subject in the video at the aforementioned date/time is 6’2” tall, with an error of ..."

Everything's OK ... until it's not. You've been qualified as an expert. Is your report ready for trial? What should be in a report anyway?

First off, there's two types of guidance in answering this question. The first type, people's experiences, might help. But, then again, it might not. Just because someone got away with it, doesn't make it a standard practice. Just because you've been through a few trials doesn't make your way "court qualified." These are marketing gimmicks, not standard practices. The second type, a Standard Practice, comes from a standards body like the ASTM. As opposed to the SWG's, who produce guidelines (it would be nice if you ...), standards producing bodies like the ASTM produce standards (you must/shall). For the discipline of Forensic Multimedia Analysis, there are quite a few standards which govern our work. Here's a few of the more important ones:

• E860-07. Standard Practice for Examining And Preparing Items That Are Or May Become Involved In Criminal or Civil Litigation
• E1188-11. Standard Practice for Collection and Preservation of Information and Physical Items by a Technical Investigator
• E1459-13. Standard Guide for Physical Evidence Labeling and Related Documentation
• E1492-11. Standard Practice for Receiving, Documenting, Storing, and Retrieving Evidence in a Forensic Science Laboratory
• E2825-12(17). Standard Guide for Forensic Digital Image Processing

Did your retrieval follow E1188-11? Did your preparation of the evidence items follow E860-07? Did you assign a unique identifier to each evidence item and label it according to E1459-13? Does your workplace handle evidence according to E1492-11? Did your work on the evidence items follow E2825-12?

If you're not even aware of these standards, how will you answer the questions under direct / cross examination?

Taking a slight step back, and adding more complexity, you're engaged in a forensic science discipline. You're doing science. Science has rules and requirements as well. A scientist's report, in general, is structured in the same way. Go search scientific reports and papers in Google Scholar or ProQuest. The contents and structure of the reports you'll find are governed by the accredited institution. I've spent the last 8 years working in the world of experimental science, conducting experiments, testing data, forming conclusions, and writing reports. The structure for my work was found in the school's guidance documentation and enforced by the school's administrative staff.

How do we know we're doing science? Remember the NAS Report? The result of the NAS Report was the creation of the Organization of Scientific Area Committees for Forensic Science about 5 years ago. The OSAC has been hard at work refining guidelines and producing standards. Our discipline falls within the Video / Image Technology and Analysis (VITAL) Subcommittee. In terms of disclosure, I've been involved with the OSAC since it's founding and currently serve as the Video Task Group Chair within VITAL. But, this isn't an official statement by/for them. Of course, it's me (as me) trying to be helpful, as usual. :)

Last year, an OSAC group issued a new definition of forensic science that can be used for all forensic science disciplines. Here it is:

Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context. Source: A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence. OSAC Task Group on Digital/Multimedia Science. 2017

What is a trace? A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system.

Thus it is that we're engaged in science. Should we not structure our reports in the same way, using the available guidance as to how they should look? Of course. But what would that look like?

Let's assume that your report has a masthead / letterhead with your/your agency's name and contact information. Here's the structure of a report that (properly completed) will conform to the ASTM standards and the world of experimental science.

Administrative Information
     Examiner Information
     Requestor Information
     Unique Evidence Control Number(s)
     Chain of Custody Information
Summary of Request
     Service Requested (e.g. photogrammetry, authentication, change of format, etc.)
Methodology
     Equipment List
     Experimental Design / Proposed Workflow
Limitations / Delimitations
     Delimitations of the Experiment
     Limitations in the Data
     Personnel Delimitations / Limitations
Processing
Amped FIVE Processing Report can be inserted here as it conforms to ASTM 2825-12(17).
Results / Summary
     Problems / Errors Encountered
     Validation
     Conclusions
     List of Output File(s) / Derivatives / Demonstratives
Approval(s)
     Examiner
     Reviewer
     Administrative Approval

It would generally conclude with a declaration and a signature. Something like this, perhaps:

I, __________, declare under penalty of perjury as provided in 28 U.S.C. §1746 that the foregoing is true and correct, that it is made based upon my own personal knowledge, and that I could testify to these facts if called as a witness.

Now, let's talk about the sections.

The Administrative section.

• You're the examiner. If you have help, or someone helped you in your work, they should be listed too. Co-workers, subcontractors, etc.
• The requestor is the case agent, investigator, or the client. The person who asked you to do the work.
• Every item of evidence must have a unique identifier.
• Every item received must be controlled and it's chain of custody tracked. If others accessed the item, their names would be in the evidence control report / list. DEMS and cloud storage solutions like Evidence.com can easily do this and produce a report.

Summary of Request

• What was it that you were asked to do, in plain terms. For example, "Given evidence item #XXX, for date/time/camera, I was asked to determine the vehicle's make/model/year" - comparative analysis / content analysis. Or, "Given evidence item #XXX, for date/time/camera, I was asked to estimate the unknown subject's height" - photogrammetry. Or, "Given image evidence item #XXY-X, retrieved from evidence item #XXY (see attached report), I was asked to determine if the image's contextual information had been altered" - authentication.  
• Provide an abstract of the test and the results - a brief overview of what was done and what the results were (with references to appropriate page numbers). 

Methodology

• What tools did you use - hardware / software? You may want to include a statement as to each and their purpose / fitness for that purpose. As an example, I use Amped Five. Amped Five is fit for the purpose of conducting image science experiments as it is operationalized from peer-reviewed / published image science. It's processing reports include the source documentation. 
• Your proposed workflow. What will guide your work? Can you document it easily? Does your processing report follow this methodology? Hint, it should. Here's my workflow for Photogrammetry, Content Analysis, and Comparative Analysis. You can find it originally in my book, Forensic Photoshop. It's what I use when I work as an analyst. It's what I teach.

Limitations / Delimitations

• Delimitations are the bounds within which your work will be conducted. I will test the image. I won't test the device that created the image.
• With DME, there are a ton of limitations in the data. If the tested question is, what is license plate, and a macro block analysis determines that there is no original data in the area of the license plate, then that is a limitation. If the tested question is, what is the speed of the vehicle, and you don't have access to the DVR, then that is a huge limitation. Limitations must be stated.
• Personnel issues should also be listed. Did someone else start the work that you completed? Was another person employed on the case for a specific reason? Did something limit their involvement? If the question involves the need to measure camera height at a scene, and you can't climb a ladder so you mitigated that in some way, list it. 

A side note here ... did you reach out to someone for help? Someone like the DVR's technician or the manufacturer of your analysis tool's support staff? Did they assist you? Make sure that you list their involvement. Did you send out a copy of the evidence to someone? If yes, is it within your agency's policy to release a copy of the evidence in the way that you've done so for the case? As an example, you send a still image of a vehicle to the FVA list asking for help. You receive a ton of advice that helps you form your conclusion, or helps the investigation. Did you note in your report that you interacted with the list and who helped? Did you provide a copy of the correspondence in the report package? Did you provide all of the responses or just the ones that support your conclusion? The ones that don't support your eventual conclusion should be included, with an explanation as to why you disagree. They're potentially exculpatory, and they should be addressed.

Remember, on cross examination, attorneys rarely ask questions of people blindly. They likely already know the answer and are walking your down a very specific path to a very specific desired conclusion.  Whilst an attorney might not subpoena Verint's tech support staff / communications, as an example, they may have access to the FVA list and may be aware of your communications about the case there. You may not have listed that you received help from that source, but the opposing counsel might. You won't know who's watching what source. They may ask if you've received help on the case. How would you answer if you didn't list the help and disclose the communications, all of the communications? If your agency's policy prohibits the release of case related info, and you shared case related info on the FVA list, your answer to the question now involves specific jeopardy for your career. I've been assigned to Internal Affairs, I've been an employee rep, I know how the system works when one has been accused of misconduct. How do you avoid the jeopardy? Follow your agency's policies and keep good records of your case activity.

Processing

• These are the steps performed and the settings used. This section should read like a recipe so that some other person with similar training / equipment can reproduce your work. This is the essence of Section 4 of ASTM 2825. Amped FIVE Processing Report can be inserted here as it conforms to ASTM 2825-12(17). 

Results / Summary

• Did you encounter any problems or errors. List them.
• How did you validate your results? Did anyone peer review your work? This can include test/retest or other such validity exams.
• Conclusions - your opinion goes here. This is the result of your test / experiment / analysis.
• List of Output File(s) / Derivatives / Demonstratives

Approval(s)

• Examiner (your name here), along with anyone else who's work is included in the report.
• Reviewer(s) - was your completed work reviewed? Their name(s).
• Administrative Approval - did a supervisor approve of the completed exam?

Do your reports look like this? Does the opposing counsel analyst's report look like this? If not, why not? It may be an avenue to explore on cross examination. It's best to be prepared.

I know that this is a rather long post. But, I wanted to be rather comprehensive in presenting the topic and list the sources for the information listed. Hopefully, this proves helpful.

Enjoy.

Strengthening Forensic Science in the US

The Organization of Scientific Area Committees (OSAC) began sending out letters to subject matter experts a few weeks ago. OSAC is part of an initiative by NIST and the Department of Justice to strengthen forensic science in the United States.

Of particular interest to the readers of this blog, the IT/Multimedia Scientific Area Committee (SAC) has named and published its committee Chairs. The IT/Multimedia SAC consists of the Speaker Recognition, Imaging Technologies, Digital Evidence, and Facial Identification subcommittees.

I've been telling you that this is coming since the NAS report was published a few years ago. Well. Here it is.

The next press release from the OSAC should contain the names of the subcommittee members as well as the dates/locations for the first SAC meetings.

Enjoy.

OSAC Subcommittee on Imaging Technologies

The OSAC subcommittees have announced their membership rosters. The Imaging Technologies Subcommittee's list can be found here. It's my privilege and quite the honor to be selected to serve on this subcommittee. I'm sure that there'll be a ton of work to do.

Review: Forensic science. The importance of identity in theory and practice

A paper was recently published over on Science Direct (link) exploring what may be a "crisis" in forensic science. Here's the abstract of the paper:

"There is growing consensus that there is a crisis in forensic science at the global scale. Whilst restricted resources are clearly part of the root causes of the crisis, a contested identity of forensic science is also a significant factor. A consensus is needed on the identity of forensic science that encompasses what forensic science ‘is’, and critically, what it is ‘for’. A consistent and cogent identity that is developed collaboratively and accepted across the entire justice system is critical for establishing the different attributes of the crisis and being able to articulate effective solutions. The degree to which forensic science is considered to be a coherent, interdisciplinary yet unified discipline will determine how forensic science develops, the challenges it is able to address, and how successful it will be in overcoming the current crisis."

The article seems an exploration of struggle for identity, as the title suggests. What is it? What are we to do with it? What happens when it's not done correctly? Who's responsible for reform? Curiously, even as the paper notes the work done on forensic science in the US, it omits reference to the 2017 A Framework for Harmonizing Forensic Science Practices and Digital/Multimedia Evidence (link). I point this out because the OSAC's document provides a solid definition of forensic science that can serve as a foundation from which to explore the paper's topic, as well as to provide a path forward for research and practice.

As a reminder, the definition of forensic science, "The systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context." "A trace is any modification, subsequently observable, resulting from an event."

I think that within the OSAC's definitions, the goals outlined in the paper can be achieved.

Nevertheless, the likely root of the "crisis" as observed by the author can be seen in another area altogether - bias.

The topic of bias, and the many places it influences the justice process has been explored in depth. Particularly, these two papers (link) (link) explore the impact of bias from the standpoint of each of the stakeholders and the influence of modern media.

Another place where bias occurs is in the selection and continuation of cases by the prosecution. For example, what effect does linking a prosecutor's "win / loss" record to their promotability within their organization have on their decision making process? Once they've filed a case, is there evidence of an "escalation of commitment" when confronted with problems in proving their case? With "escalation of commitment" bias, the prosecutor may seek out "fringe techniques" in an attempt to support their theories of the case. Is it these "fringe techniques," not forensic science, that have contributed to the observed "crisis" - as was illustrated in Netflix's Exhibit A?

Still, I'm glad to see that people are starting to identify that there might be a problem in the practice of "forensic science." How that problem, or "crisis," is addressed will make all the difference in the world.

Have a good weekend my friends.

Welcome

Welcome to the Forensic Multimedia Analysis blog (formerly the Forensic Photoshop blog).

With the latest developments in the analysis of multimedia (video, audio, images, and metadata), we move the discussion beyond a single piece of software to include (in no particular order) processing and analysis fundamentals, court cases, upcoming training offerings, product reviews, current research, standards and practices, industry events and trends, and much more.

Digital / multimedia forensic analysis covers the domains of:

• Authentication
• Photogrammetry
• Photographic Comparison
• Photographic Content Analysis

Clarification, enhancement, and restoration are processes that can occur within the domains, but aren't domains in and of themselves.

We use the term digital / multimedia forensic analysis, as opposed to forensic video analysis or forensic audio analysis as we acknowledge that modern multimedia evidence potentially contains audio, images, video, as well as metadata. Thus, we need to be able to process and analyze everything that's found in the evidence files that we receive.

It's also important to define "forensic science." For this, I'll refer to "A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence." OSAC Task Group on Digital/Multimedia Science. 2017 (link): "Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context."

What is a trace? "A trace is any modification, subsequently observable, resulting from an event." You walk within the view of a CCTV system, you leave a trace of your presence within that system. You send a text, you leave a trace on your phone.

Within this framework, and wherever possible, we'll frame our discussion around standards and science. Validity, reliability, and reproducibility will be our goals ... not to present something unique and proprietary that only we can do here, but to illustrate the science behind the tools and techniques so that you can do it too.

I hope you enjoy your time here.

Jaime

The need for objective measures in forensic evidence

The latest edition of Significance, the journal of the Royal Statistical Society, features an article on the importance of objective measures and standards in evaluating evidence. Whilst many of the points made by the author are valid and applicable, practitioners within the digital / multimedia forensic sciences already have standards and guidelines for basing their work on an objective footing.

Investigators want to use video evidence to "identify" an object in a video - gun, knife, etc. But this "identification" is often complicated by a lack of nominal resolution in the region of interest. Many will simply state that the object has been identified, without actually taking the steps to formally conclude (prove) their assertions. They go with what they "know" (abductive reasoning), as opposed to what they can prove. There's a better way.

First off, here are some relevant definitions from the SWGDE Digital & Multimedia Evidence Glossary. Version: 3.0 (June 23, 2016)

Artifact: A visual/aural aberration in an image, video, or audio recording resulting from a technical or operational limitation. Examples include speckles in a scanned picture or “blocking” in images compressed using the JPEG standard.

Image Analysis: The application of image science and domain expertise to examine and interpret the content of an image, the image itself, or both in legal matters.

Image Comparison (Photographic Comparison): The process of comparing images of questioned objects or persons to known objects or persons or images thereof, and making an assessment
of the correspondence between features in these images for rendering an opinion regarding identification or elimination.

Image Content Analysis: The drawing of conclusions about an image. Targets for content analysis include, but are not limited to: the subjects/objects within an image; the conditions under which, or the process by which, the image was captured or created; the physical aspects of the scene (e.g., lighting or composition); and/or the provenance of the image.

Nominal resolution: The numerical value of pixels per inch as opposed to the achievable resolution of the imaging device. In the case of flatbed scanners, it is based on the resolution setting in the software controlling the scanner. In the case of digital cameras, this refers to the number of pixels of the camera sensor divided by the corresponding vertical and horizontal dimension of the area photographed.

Video Analysis: The scientific examination, comparison, and/or evaluation of video in legal matters.

It's also important to define "forensic science." For this, I'll refer to "A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence." OSAC Task Group on Digital/Multimedia Science. 2017: "Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context."

What is a trace? "A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system."

SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) provides a few more definitions.

Class Characteristic: A feature of an object that is common to a group of objects.

Individualizing Characteristic: A feature of an object that contributes to differentiating that object from others of its class.

With the definitions in mind, SWGDE Best Practices for Image Content Analysis. Version: 1.0 (February 21, 2017) provides the framework to begin the Content Triage step in the workflow - can I answer the question with the evidence file?

5. Evidence Preparation
5.2 Based on the request for examination, determine if submitted imagery is available to complete requested analysis. Determine whether submitted imagery is of sufficient quality to complete the requested examination, or if the image quality will have an effect on the degree to which an examination can be completed.
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.

(do I have sufficient nominal resolution within the target area to fulfill the request - answer the question? The nominal resolution, measured in pixels per inch or inches per pixel - or it's metric equivalent, is the objective measure.)

6. Examinations Method
There is no one specific methodology for content analysis. The methodology for analysis will primarily be derived to answer the requested examination. However, any methodology applied to content analysis should incorporate an analysis of the imagery, the cataloguing of relevant features, an evaluation of the significance of the detected features, an evaluation of the limiting factors of the imagery, the formation of a conclusion, and a verification of the analysis. The repeatability (and / or reproducibility) of the procedure and documentation of the workflow is of paramount importance. Documentation should be performed contemporaneously.
6.1 Assess the contents of the image, to determine whether factors are present that can answer the examination request. The examination request generally will fall into one of the following categories:
6.1.1 Analysis to determine the conditions under which, or the process by which, the image was captured or created. Examples include, but are not limited to, the limitations of the recording device, and the inclusion of artifacts based on the file format or compression. This can help to answer the question “How does the recording system affect what is visible in the scene?”
6.1.2 Analysis to determine the physical aspects of the scene, including events captured. Examples include, but are not limited to, the lighting and composition of the scene, the presence of specific objects within the scene, a determination of the interaction between objects in the scene, and a description of events within a scene. This can help to answer the questions “Is a specific object visible in the scene?” or “What happened in the scene?”
6.1.3 Analysis to determine the classification of an object within an image. Examples include, but are not limited to, the make, model, and year of a vehicle, the determination of a manufacturing logo, and the determination of the brand and model of a weapon. This can help to answer the question “What is the object visible in the scene?”
6.1.4 Analysis to determine the location or setting of the image content. Examples may include either a general setting (e.g. Portland, Oregon) or a specific setting (e.g. Conference Room 23, the Northwest Corner). This can help answer the question “Where is the scene?”
6.3 Assess the image for features that contribute to the ability to form a conclusion, and record observed features. Consider the weight or importance of identified features, in order to determine the focus of the examination. Examples of features may include logos, shapes, reflections, or specific items.

How does one move from "artifact" (6.1.1) to "object" (6.1.3)?

As regards “artifact:"

When nominal resolution is low, the analyst must be certain of the provenance of the features within the region of interest in the evidence item. Are the “items of interest” actual representations of things, or are they an “aberration” in the image, resulting from problems in compression and etc. In order to move from “a few dark blocks of pixels” is not an “artifact” to the "few dark blocks of pixels” is an “object” - the rules for classifying an object apply. Additionally, one must rule out the possibility that what one sees is not an artifact. This must be done in a systematic / coherent fashion in order to be a forensic science exercise.

As regards “object:"

In order to move from “a few dark blocks of pixels” that are not a result of an “aberration" to firearm, there needs to be Class Characteristics present in the view that would lead to the conclusion of "object." In order to form a conclusion, those Class Characteristics should be identified by the analyst - overall shape: the presence of features of an object that contributes to differentiating that object from others of its class  Is the nominal resolution in the region of interest sufficient to conclude as to class and individualizing characteristics? This is the objective measure.

The standards and guidelines are there. It is left to the investigator, the analyst, the attorney, and the Trier of Fact to follow the rules and accept the facts as they present themselves. What does the investigator do when told by the analyst that the nominal resolution does not support a conclusion? Does he / she shop around for another analyst? The investigator must not fall victim to cognitive perseverance (bias) when faced with this new information.

Additionally, these stakeholders must understand when they're on sound scientific footing or when they're just taking their best shot.

Types of Resoning

Abductive reasoning: taking your best shot
Deductive reasoning: conclusion guaranteed
Inductive reasoning: conclusion merely likely

What most in the forensic sciences don't quite acknowledge is that they operate in the world of abductive reasoning most of the time.

From a US Constitution standpoint, from the standpoint of Constitutional Policing, it is for the side offering an item as “proof” of some condition (e.g., object = firearm) to actually prove class = firearm. It perverts the cause of justice to require the defense to prove that it isn’t.

This is the level of detail that we get into when I present training in the digital / multimedia forensic sciences, as well as my other educational offerings. If you need help on a case, or want a second opinion, feel free to contact me today.

What is Analysis?

What is analysis?

a·nal·y·sis [əˈnaləsəs] - NOUN
     analyses (plural noun)

• detailed examination of the elements or structure of something. "statistical analysis" · "an analysis of popular culture"

synonyms: examination · investigation · inspection · survey · scanning · study · scrutiny · perusal · exploration · probe · research · inquiry · anatomy · audit · review · evaluation · interpretation · anatomization

• the process of separating something into its constituent elements. Often contrasted with synthesis. "the procedure is often more accurately described as one of synthesis rather than analysis"

synonyms: dissection · assay · testing · breaking down · separation · reduction · decomposition · fractionation
antonyms: synthesis

Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context.  (Source: A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence. OSAC Task Group on Digital/Multimedia Science. 2017)

What is a trace? A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system.

Thus, forensic video analysis (or forensic multimedia analysis) can be seen as a systematic and coherent examination of video (multimedia) traces (elements) to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context.

In the former definition, we can see the quantitative nature of analysis. The latter definition reveals it's potential qualitative elements.

In a quantitative data analysis, things are stable, controlled - facts can be obtained (facts are measurable / objective). In a qualitative data analysis, things are dynamic. Your role as an observer may influence the analysis. What is "true" depends on the situation & setting (truths are things we "know" - subjective). A quantitative study is controlled. A qualitative study is observed.

A qualitative study's purpose is to describe or understand something. The purpose of a quantitative study is to test, resolve, or predict something (e.g. in order to use a DVR to determine speed of an object within it's derivative video files - results will be a range of values, one must resolve how the DVR creates files "typically" through a controlled series of tests).

The analyst's viewpoint during a quantitative study is logical, empirical, deductive (conclusion guaranteed). In a qualitative study, it's situational and inductive (conclusion merely likely) or abductive (taking one's best shot). Performing a comparative analysis with convenience samples is an example of taking one's best shot. A quantitative study would feature an appropriate sample size calculation and note any limitations that arose as a result of not being able to achieve the appropriate samples.

From a contextual standpoint, a quantitative's context is not taken into consideration but rather controlled via methodological procedures. In this way, potential bias is mitigated. In a qualitative study, context matters - values, feelings, opinions, individual participants matter.

In a quantitative study, the analyst seeks to solve, to conclude, or to verify a predetermined hypothesis. With a qualitative study, the orientation changes - seeking rather to discover or explore. This can occur often in investigations - new information developed leads to changes in the direction of the investigation as things / people are ruled-in / ruled-out.

In a quantitative analysis, the inputs and results are numerical - data is in the form of numbers / numerical info. A qualitative analysis is narrative in nature - data is in the form of words, sentences, paragraphs, notes, or  pictures / graphics / etc.

After conducting a quantitive analysis, one's results / findings can be generalized to other populations or situations. The results of a qualitative analysis are case specific, particular, or specialized.

With all of this in mind, what is analysis? What type of analysis are you conducting? What type of analysis are you reporting? When analyzing the work of other analysts, what type of work are they conducting / reporting?

You can use this dialog to build a template / matrix. In reviewing work, examine the elements above to determine if the work is quantitative or qualitative. For example, you're reviewing an analyst's work in on a measurement request (photogrammetry). The results section features a picture that has been marked up with arrows and text. No methodology is discussed. These results would be considered qualitative. If the results section featured a conclusion, a range of values, error estimation, and a reference / methodology section, it could be considered quantitative. You could take the analyst's data and reproduce their study - which is not possible from an annotated picture.

The elements for a quantitative analysis described above, when reported back to the Trier of Fact, help ensure that you've maintained standards compliance (ASTM E2825-18). Rhetorical or narrative statements are fine for the introductory section of your report - a summary of the request - but are not sufficient for supporting a conclusion or describing one's processes.

If you'd like to know more, join me in an upcoming training session. For more information or to sign up, click here.

Why do you need science?

An interesting morning's mail. Two articles released overnight deal with forensic video analysis. Two different angles on the subject.

First off, there's the "advertorial" for the LEVA / IAI certification programs in the Police Chief Magazine.

The pitch for certification was complicated by this image:

The caption for the image further complicated the message for me: "Proper training is required to accurately recover or enhance low-resolution video and images, as well as other visual complexities."

Whilst the statement is true, do you really believe that the improvements to the image happened from the left to the right? Perhaps, for editorial purposes, the image was degraded, from the original (R) to the result (L). If I'm wrong about this, I'd love to see the case notes and the specifics as to the original file. Can you imagine such a result coming from the average CCTV file? Hardly.

Next in the bin was an opinion piece in the Washington Post's Radley Balko - "Journalists need to stop enabling junk forensics." It's seemingly the rebuttal to the LEVA / IAI piece.

Balko picks up where the ProPublica series left off - an examination of the discipline in general, and Dr. Vorder Brugge of the FBI in particular. It's an opinion piece, and it's rather pointed in it's opinion of the state of the discipline. Balko, like ProPublica, has been on this for a while now (here's another Balko piece on the state of forensic science in the US).

I don't disagree with any of the referenced authors here. Not one bit. Jan and Kim are correct in that the Trier of Fact needs competent analysts working cases. Balko is correct in that the US still rather sucks at science. That we suck as science was the main reason the Obama administration created the OSAC and the reason Texas created it's licensing scheme for analysts.

Where I think I disagree with Jan and Kim is essentially a legacy of the Daubert decision. Daubert seemingly outsourced the qualification process to third parties. It gave rise to the certification mills and to industry training programs. Training to competency means different things to different organizations. For example, I've been trained to competency on the use of Amped's FIVE and Authenticate. But, none of that training included the underlying science behind how the tool is used in case work. For that, I had to go elsewhere. But, Amped Software, Inc, certified me as a user and a trainer of the tools. That (certification) was just a step in the journey to competency, not the destination.

Balko, like ProPublica, notes the problems with pattern evidence exams. Their points are valid. But, it doesn't mean that image comparison can't be accomplished. It does mean that image comparisons should be founded in science. One of those sciences is certainly image science (knowing the constituent parts of the image / video and how the evidence item was created, transmitted, stored, retrieved, etc. But another one of the sciences necessary is statistics (and experimental design).

As I noted in my letter to the editor of the Journal of Forensic Identification, experimental design and statistics form a vital part of any analysis. For pattern matching, the evidence item may match the CCTV footage. But, would a representative sample of similar items (shirts, jeans, etc) also match? Can you calculate probabilities if you're unaware of the denominator in the function (what's the population of items)? Did you calculate the sample size properly for the given test? Do you have access to a sample set? If not, did you note these limitations in your report? Did these limitations inform your conclusions?

Both LEVA and the IAI have a requirement for their certified analysts to seek and complete additional training / education towards eventual recertification. This is a good thing. But, as many of us know, there are only so many training opportunities. At some point, you kind of run out of classes to take is a common refrain. Whilst this may be true for "training" (tool / discipline specific), this is so not true for education. There are a ton of classes out there to inform one's work. The problem there becomes price / availability. This price / availability problem was the primary driver behind my taking my Statistics class out of the college context and putting it on-line as micro learning. My other classes from my "curriculum in a box" concept will roll out later this year and into the next year.

So to the point of the morning's articles - yes, you do need a trained / educated analyst. Yes, that analyst needs to engage in a scientific experiment - governed both by image science as well as experimental science. Forensic science can be science, if it's conducted scientifically. Otherwise, it becomes a rhetorical exercise utilizing demonstratives to support it's unreproducible claims.

Taking the AVFA Certification Exam

I have to say that signing up and taking the AVFA certification exam was a rather pleasant experience. Yes, that seems a bit odd to say. But, it's true. It should be true that an organization treats you like an adult, respectfully walking you through the process of engaging their services.

I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.

You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.

I can't emphasize this enough, I paid $120 in fees to take this certification exam.

If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.

For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.

At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.

Back to the experience.

I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.

The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.

The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.

In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certificate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.

Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.

Coming Out of Retirement for This One: On AI, Authorship, and the Erosion of Ground Truth

It’s been a while since I posted here.

I’ve moved on from daily forensic casework and stepped back from the blog to focus on broader systems work—on justice, equity, neurodiversity, and how we reckon with broken epistemologies. But sometimes something happens that drags you out of retirement whether you like it or not. And this is one of those moments.

The Electronic Frontier Foundation (EFF) just released a thorough, scathing analysis of Axon’s “Draft One”—an AI tool currently being used in police departments to automatically generate narrative reports from body-worn camera audio. The headlines focus on missing drafts and vanishing accountability, but from where I sit, this story goes much deeper. It cuts to the core of what forensic science is supposed to be—and the mechanisms by which truth, evidence, and responsibility are disappearing in real time.

This is not just an engineering failure. It’s a perversion of process, designed to undermine transparency and suppress ground truth. And unless we start shouting about it now, loudly and persistently, we may find the damage done before it’s ever tested in court.

So here I am. Let’s talk.

---

What Is Forensic Science, Again?

When I was serving on the OSAC Video/Imaging Technology and Analysis Task Group, we spent a fair bit of time just trying to define our terms. “Forensic science” may sound straightforward, but once you include all disciplines—from biology to acoustics to digital traces—you need language that holds across those domains.

The final agreed definition read (it's probably changed since then):

Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context.

Let’s break that down.

• Systematic and coherent: It means you use a validated methodology. You test. You document. You repeat. You don’t just guess.

• Study of traces: Not assumptions. Not vibes. Not hallucinated context from a black box algorithm.

• For a legal context: Which means your findings must be discoverable, explainable, and reproducible in court.

Any tool—human or machine—that doesn’t meet those standards doesn’t belong in forensic science or in the justice system.

And yet Axon’s Draft One is being used to generate police reports that:

• Can’t be traced to an identifiable author.

• Can’t be audited for edits or AI influence.

• Can’t be cross-examined in court, because the drafts no longer exist.

That’s not justice. That’s marketing.

---

The Death of the Draft

Let’s be clear: deleting original drafts isn’t some accidental oversight—it’s a deliberate design choice. A senior Axon product manager is on record stating that storing drafts would only “create more disclosure headaches” for their customers and attorneys.

In other words, the system is intentionally structured to evade scrutiny.

In civil proceedings, experts operate under strict obligations—FRE Rule 26 (or its UK counterparts) requires comprehensive disclosure: your notes, your methodology, your tools, your working files. Everything from the History Log in Photoshop to the process report in Amped FIVE is considered part of the evidentiary record and must be turned over. This ensures transparency and allows opposing parties to challenge the foundation of any claim.

But there’s no equivalent to Rule 26 in criminal procedure.

And that absence is where much of the injustice festers. In criminal cases—especially those resolved through plea deals—discovery is often minimal, accountability mechanisms are weaker, and tools like Draft One can be quietly implemented with no real oversight. The system permits it not because it’s just, but because it’s expedient.

When a narrative is generated by AI, lightly skimmed or rubber-stamped by an officer, and submitted without retaining the drafts, who’s actually attesting to authorship?

No one.

And in a system already tilted against defendants, that’s not just a technical gap—it’s a structural abuse.

---

Officers of the Court, Without Responsibility

When a police officer submits a report, they do so as an officer of the court. That’s not metaphorical. They’re bound by law and professional ethics to tell the truth, to the best of their knowledge, under penalty of perjury.

The legal assumption is that the report is theirs. They saw the event. They wrote it down. They stand by it.

But if Draft One creates a first draft—based on audio the officer might not even remember clearly—and that draft is then lightly edited (or not at all), who is really the author?

If the report contains bias, omission, or invention, is it the fault of the AI? Or the officer? And how would you know?

EFF is absolutely right to call this a “smokescreen”—one that could allow officers to deny responsibility while still reaping the legal authority of a sworn report. It’s the worst of both worlds: automation without accountability.

---

The Market Logic Behind the Curtain

Let’s not ignore the profit motive either.

Axon is a publicly traded company. Its duty is to shareholders, not truth. Draft One is being rolled out not because it meets evidentiary standards—but because it saves time, cuts labour costs, and locks departments deeper into the Axon ecosystem.

In a context where most cases end in plea deals—where defence teams rarely get the chance to challenge reports through discovery or trial—these AI-generated narratives may never face real scrutiny. They’ll become “truth” by default, untested and untestable.

This is how you change the epistemology of the criminal legal system without ever passing a law. You let a company rewrite the workflow.

And if no one objects loudly enough, that new workflow becomes the norm.

---

The Real Problem: Validation and Ground Truth

When I wrote my 2020 article on Ground Truth in Digital/Multimedia Forensics, I was wrestling with many of these same questions.

Validation isn’t optional. You don’t get to skip the experiment just because the vendor says their tool is “reliable.” You have to test—under real conditions, with real data, using accepted scientific methods.

And if your tool doesn’t allow for that kind of testing—if it deletes its own outputs, obscures its authorship, or refuses to log its transformations—then you are not practising forensic science.

You’re engaging in speculative automation.

You’re telling stories with invisible authors and calling them evidence.

---

A Final Word (For Now)

I never thought I’d see the day when officers could submit reports they didn’t write, generated by a system that deletes its drafts, under the banner of “efficiency”—and still have those reports carry the full weight of legal authorship.

But here we are.

And so, for the moment, I’m coming out of retirement. Because the integrity of the judicial system matters. Because ground truth matters. And because the legal system cannot afford to outsource authorship and accountability to a product designed to dodge both.

This isn’t innovation. It’s abdication.

And if we don’t push back now, we may find ourselves standing in courtrooms arguing with ghosts—armed only with final reports, stripped of history, speaking with no traceable voice.

Retrieval / Seizure of Electronic Evidence from Crime Scenes

Twelve years ago, a few multimedia analysts got together via emails and phone calls to create a training class around an emerging topic - what to do with the video contained within these black boxes that folks were encountering at crime scenes.

Several of us had "figured it out." We figured out what worked in our own contexts. What worked for me in Los Angeles was slightly different than what worked in New York, or Virginia, or Texas.

We hashed out a loose curriculum for what was eventually delivered at the 2007 NaTIA conference in Pittsburgh. In that course, we presented a hands-on overview of the technology, the legal aspects, and ideas for procedures that could easily be implemented by state / local agencies.

This experience lead to the creation of the red flip book, aka Best Practices for the Retrieval of Video Evidence from Digital CCTV Systems. I'm quite proud of what we all accomplished in the production of that handy guide, which is still available from the CCTSO.

From there, the group was brought together again to create the curriculum for what would become the RECVR class at FLETC.

From that point, several of us went on to create separate versions of the essential materials, delivering it across the county (and the world). I've had the privilege of presenting this topic across the US and Canada, as well as in Pretoria, South Africa to the South African Police Service.

Whilst most classes need to choose a platform from which to work, and you'll need tools to perform essential work in restoration, clarification, and analysis, my curriculum has always attempted to offer a "non-platform specific" option to learners. I've presented as many tools as possible, many of them free / open source. I've focussed on policy, procedure, and workflow. I've split the topics around retrieval from the topics around seizure. I think the curriculum is rather comprehensive, yet accessible to learners of all types (novices - advanced analysts).

During my years with Amped, there was a lot of internal tension around those classes that I had previously designed and presented being offered by an Amped employee. Thus, they weren't. There was also a bit of angst about the way in which I presented the training around the Amped tools. I don't like simply focusing on buttons and functions. I try to contextualize the tools, and thus present the training in a way that builds competency.

So, with Amped's retreat, I'm happy to say that my full slate of classes is back on offer. You can find them on all listed by clicking here. Retrieval / Seizure of Electronic Evidence from Crime Scenes is back and better than ever. It doesn't really compete directly with the other classes out there. On the contrary, it's entirely complimentary.

You see, every instructor and curriculum developer has their own point of view and experiences. Each brings all of that to their courses. I'm no different in that respect. I don't usually toot my own horn, but this is different. My story helps frame the creation of the classes. Here it is:

• I'm a Certified Audio/Video Forensic Analyst (AVFA). The process of investigating the three bodies that offer certification, honestly apprising the costs/benefits of each, deciding on a path, and carrying it out to completion has been an over-a-decade journey.
• I'm the author of the best-selling book Forensic Photoshop (available on Amazon.com). Getting a book published isn't as easy as it seems. Ten years ago, it was a daunting task. Putting yourself out there for others to critique is an amazingly stressful ordeal. I'm glad that I did it.
• I'm a co-author of Best Practices for the Retrieval of Video Evidence from Digital CCTV Systems (red flip book). Collaborating with an amazing group of international all-stars in the field, listening to their diverse views, creating, compromising, standing firm, and finally seeing one's work in printed form was an awesome experience.
• When I retired from the LAPD, I had spent a third of my life in police service. Based on an 8 hour work day, and the amount of actual hours I had worked, I had worked 33 years worth of 8 hour days during that time. During the last half of my career,  I hardly saw my family and friends, my health suffered, and I was heading towards burn-out. Given the amount of crime in LA, and the relatively small size of my old group, a lot of the way in which I work and structure my courses comes from the "factory of crime" and the need to work efficiently in the face of constant requests for service. Also, my focus on human factors, taking care of one's self, and doing one's best to eliminate bias comes from this experience.
• I currently serve the community on the Organization of Scientific Area Committees on Forensic Science (OSAC), Video / Image Technology and Analysis (VITAL), as the Video Task Group Chair. Last year, we moved the DCCTV Guide to an ASTM standard. This year, we're about ready to move the many SWG docs around forensic video analysis to a common ASTM standard.

All of this finds its way into my classes. I'm not just a talking head. I'm not just reading a script. I'm giving my honest assessment of the topic, and a helpful set of instructions to get the learner to mastery.

If you're ready to move beyond "button pushing," head over to Apex Partners to check out what's on offer. Sign up for a class or two. I hope to see you soon.

Changing times

I've been in the "video forensics" business for quite some time now. I've seen enough to notice trends in the industry. I've seen people come and go. Today, I want to comment on a coming trend that I believe will impact everyone in the business, LEOs and privateers alike.

Here's what I mean.

Going back to about 2006, the economy was booming and folks were happy. Then 2007 hit and the economy tanked. As belts tightened, people cut back on entertainment and other non-essential things. A result of this was major cut-backs in the movie business. Many out of work editors and producers entered the business of video forensics. They guessed that because of their knowledge of the tools - Avid MC, PremierePro, Final Cut, etc - they could go out there and compete for work, offering their services and "expertise" in video to the courts, attorneys, PIs, and the like. There were few success stories and a lot of colossal fails. Very few of these folks are still around.

Another trend is emerging.

In the push to assure future success, parents have been steering their kids to STEM degrees. Many have pursued and achieved doctorates in the STEM fields only to find that there is a glut of people on the market with such degrees (in my academic field, there's about a 600/1 ratio of applicants to jobs/grants). Some are leaving their degree field, using their expertise in experimental design and statistics (gained by every PhD) in a variety of useful ways (Think Moneyball).

A case* from Arizona last year serves as the canary in the video forensics coal mine. It's a firearms case, but all the issues can easily be applied to our field. In State v Romero (2016), the Arizona Supreme Court said that the trial court erred in not allowing the defense to call their "expert." The person in question wasn't a firearms examiner or a tool-mark examiner. He is an expert in Experimental Design, with a PhD in the discipline.

Here's some relevant parts of the ruling:

"...Dr. Haber was not offered to testify whether Powell had correctly analyzed the toolmarks on the shell casings. Instead, Dr. Haber, based on his expertise in the broader field of experimental design, criticized the scientific reliability of drawing conclusions by comparing tool marks."

"...Arizona Rule of Evidence 702 allows an expert witness to testify if, among other things, the witness is qualified and the expert’s “scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence . . . .” Trial courts serve as the “gatekeepers” of admissibility for expert testimony, with the aim of ensuring such testimony is reliable and helpful to the jury."

Hint, every state court and the US federal courts have a similar rule governing expert witnesses and their testimony.

"... The trial court here concluded that Dr. Haber was not qualified to testify as an expert in firearms identification. In affirming, the court of appeals noted that Dr. Haber, although having reviewed the literature on firearms identification, had not previously been retained as an expert on firearms identification, conducted a toolmark analysis, attempted to identify different firearms, or conducted research on firearms identification. 236 Ariz. at 458 ¶¶ 23-25, 341 P.3d at 500."

"... The issue, however, is not whether Dr. Haber was qualified as an expert in firearms identification, but instead whether he was qualified in the area of his proffered testimony — experimental design. Here, the trial court determined that Powell was qualified to offer an expert opinion that the shell casings were all fired from the same Glock. But Romero did not offer Dr. Haber as an expert in firearms identification to challenge whether Powell had correctly performed his analysis or formed his opinions. Instead, Dr. Haber’s testimony was proffered to help the jury understand how the methods used by firearms examiners in performing toolmark analysis differ from the scientific methods generally employed in designing experiments."

Did you catch that? Dr. Haber was retained to challenge the validity of the method used in the prosecution's examination - to illustrate "... how the methods used by firearms examiners in performing toolmark analysis differ from the scientific methods generally employed in designing experiments."

"... Under Rule 702, when one party offers an expert in a particular field (here, the State’s presentation of Powell as an expert in firearms identification) the opposing party is not restricted to challenging that expert by offering an expert from the same field or with the same qualifications. The trial court should not assess whether the opposing party’s expert is as qualified as — or more convincing than — the other expert. Instead, the court should consider whether the proffered expert is qualified and will offer reliable testimony that is helpful to the jury.  Cf. Bernstein, 237 Ariz. at 230 ¶ 18, 349 P.3d at 204 (noting that when the reliability of an expert’s opinion is a close question, the court should allow the jury to exercise its fact-finding function in assessing the weight and credibility of the evidence)."

"... The gist of Dr. Haber’s proffered testimony was that the methods generally used in conventional toolmark analysis fall short of scientific standards for experimental design. Dr. Haber’s testimony was therefore directed at the scientific weight that should be placed on the results of Powell’s tests. Such questions of weight are emphatically the province of the jury to determine. E.g., State v. Lehr, 201 Ariz. 509, 517 ¶¶ 24–29, 38 P.3d 1172, 1180 (2002). "

"... Apart from Dr. Haber’s qualifications, his testimony would not have been admissible unless it would have been helpful to the jury in understanding the evidence. Ariz. R. Evid. 702(a). The State presented Powell’s testimony that the indentations on shell casings demonstrated that the Glock had fired all the shells, including those at the murder scene, and the State argued that the toolmark comparisons demonstrated a match to “a reasonable degree of scientific certainty.” Dr. Haber’s testimony would have been helpful to the jury in understanding how the toolmark analysis differed from general scientific methods and in evaluating the accuracy of Powell’s conclusions regarding “scientific certainty.”"

"... The thrust of Dr. Haber’s testimony was that the methods underlying toolmark analysis (here comparing indentations and other marks on shell casings) are not based on the scientific method, but instead reflect subjective determinations by the examiner conducting the analysis. Haber would have explained that unlike experts who use other forms of forensic analysis rooted in the scientific method, firearms examiners do not follow an accepted sequential method for evaluating characteristics of fired shell casings and comparing them to control subjects. By describing the methods used by toolmark examiners, Dr. Haber’s testimony could have helped the jury assess how much weight to place on Powell’s “scientific” conclusion that the shell casings at the murder scene could only have been fired from the Glock found by the police when they stopped Romero." How big was the sample size in your experiment? How did you determine the appropriateness of that size? How did the casing's markings compare to a normal distribution of values derived from the sample / control subjects?

"... One of his critiques of the methodology used by firearms examiners is that they do not employ identifiable, standardized protocols." Show me the peer-reviewed, published source that describes the method used.

"... Dr. Haber’s testimony was intended to highlight that the conclusions drawn by firearms examiners from toolmarks do not result from the application of articulable standards and lack typical safeguards of the scientific method such as independent verification by other examiners. Thus, Dr. Haber’s testimony could have helped the jury to understand any eficiencies in the experimental design of toolmark analysis and to assess any suggestion that such analysis was “scientific.” Cf. Salazar-Mercado, 234 Ariz. at 594 ¶ 15, 325 P.3d at 1000 …" Who checked your work and signed-off on it? 

So why such a long post? I saw a video over on Deutsche Welle called "Crime fighting with video forensics." In it, the featured person made this statement: “each vehicle has a unique headlight spread pattern." Does it now? How does he know this? Did he conduct a study? Where is it published? Has he every been asked to prove out his methodology? What was the sample size of the experiment? How was the appropriate size for the sample calculated? How would his "headlight spread pattern" methodology stand up to cross-examination by an attorney prepared by someone with knowledge of experimental design? Remember, there are a lot of out-of-work PhDs out there? What would happen if Dr. Haber was the opposing expert in your case?

The Reddit Bureau of Investigation tackles the subject here. A link from that page contains the following quote, "... all the things your describing sound almost.... Imperfect? I mean, it scares me to think I might get pinned for a crime because I have a similar headlight spread as someone else … So what I'm asking is, are techniques like headlight spread and clothing identification taken very seriously in court? ..." According the the DW story, the matching of the "headlight spread pattern" did lead to a conviction in the highlighted case. The posts are about 5 years old. Plenty of time for someone to actually test this method and publish results - not just post questions on Reddit. But, I can't find any studies in the academic repositories.

Now, I may seem to be picking on one person. I'm not. I'm picking on the use of techniques that are called "science" but have no foundation in any science or the scientific method. I found police-led training on the subject with a simple Google search. Well-meaning folks will be exposed to this topic and begin to use it in their investigations - perhaps unaware of the challenges to it's validity that they may face if/when they testify as to their work.

Errors in conclusions and the use of untested methodologies threaten forensic science. It's not me saying this, it's the focus of the NAS Report. It's the reason the OSAC was created. If you're in the "video forensics" discipline, and you're giving your OPINION about something related to the evidence, PLEASE be sure that your opinion is grounded in valid and reliable science - science that you can quote when asked. For example, if you're using the Rule of Thirds to calculate the height of an unknown subject / object in a CCTV video, you will have problems under a capable cross examination. Where in academics / science can you find a paper that tells you how to employ this method for this purpose? Hint, you can't. If you're using Single View Metrology in your measurements, you'll easily find the source document for this technique as well as the many papers that cite this technique.

And this is where the weakness in many "analysts" work can be found. When giving your opinion, what is the source of your conclusion? Which paper? Which study? How about simply listing your references / sources in your report so there's no confusion as to the basis of your opinions?

My entry into grad school opened my eyes as to what I didn't know and what the various trade groups where I'd received my training couldn't prepare me for. My pathway to my dissertation had me laser focussed on stats, experimental design, sample sizes, validity, and defending my work in front of people who have gone down a similar path and know way more than me. It's humbling to defend one's work - to be cross-examined by such brilliant people. But, iron sharpens iron. I'm the better for it.

Rather than tell you, it'll be OK, I'm saying watch out. You're heading down an unsustainable path. If folks want to continue to use this method - "headlight spread pattern analysis," probability says that there's going to be a challenge. Do you want that to be you? Are you prepared for it?

Something to think about ...

*I'm not an attorney. This is not legal advice. This is not about one person or one case, but the use of untested / un-scientific techniques. Check your six. Relax. Breathe. Love.