Wednesday, July 10, 2019

What is "official training?"

Years ago, sitting in LEVA's old class, Forensic Video Analysis and the Law, Grant Fredericks stood at the front of the class room and delivered a module on the use of Photoshop to a room full of civil servants. To the best of my knowledge, Grant is not, nor has he ever been, an Adobe Certified Expert (ACE) in Photoshop. Very few forensic video analysts are ACEs as we only utilize about 1/3 of what Photoshop can do, and the ACE test is comprehensive. Thus, the ACE isn't necessarily a good metric or barometer of a forensic video analyst's grasp of Photoshop.

Years later, sitting in another LEVA facilitated class, George Reis and Casey Caudle presented a deeper dive into Photoshop. George might be / have been an ACE, but Casey wasn't.

I have presented countless Photoshop training classes and have written a book on Photoshop for forensic analysts.

None of this was ever "official" Adobe activity. Even when I presented two days of Photoshop training in the main training room at Adobe in San Jose, it wasn't "official" Adobe training. It was Adobe being nice and gracious in their sponsorship of NaTIA, under whose banner I was teaching.

Has a lack of "official" Adobe training every been a problem for you in your testimonial experience? I've been asked about my training. I've listed it on my CV. Syllabus files are available for the courses I teach. But, case law can't be found where an analyst was excused due their having been trained in Photoshop, but not during an "official" Adobe training event.

Professional photographers and digital artists know the name, Scott Kelby. Scott has an amazing record of training photographers and artists in the intricacies of Photoshop and the various Adobe tools. He's assembled the best and brightest to his brand, offering amazing value. Prior to Beckley in California, I was a member his National Association of Photoshop Professionals. NAPP is not an "official" Adobe endeavour. It's a group of likeminded individuals, supported by Adobe. But it's Kelby's thing, not Adobe's.

Elsewhere in our tool set, countless analysts have sat through Larry Compton's FOSS tool lectures and courses. With FOSS tools, there's really no possibility of an "official" training. If you use the popular ffmpeg, did you travel to France to get training from the originators of the tool? Likely not. You may have read a bit on the net, browsed the wiki, reached out to someone like Larry for advice, then hit the command line with gusto. Are your results invalid because Larry isn't authorized to deliver training on his basket of  tools? Validity isn't a function of the "officialness" of the instruction you've received.

In the period between 2012 and early 2019, I was in the employ of Amped Software, Inc. I delivered training on Amped's product line in the US, Canada, and South Africa. I went on site to teach a single agency's staff. I hosted courses in our offices with a mix of people from over 40 countries. The curriculum that I delivered was (and still is) informed by the context in which the courses were presented.

I'm in a unique position, being a trained (CA POST) and educated (WGU MEdID) curriculum designer. The focus of my training sessions has always been "tool-assisted training to competency," as defined by the ASTM and informed by my years of experience in law enforcement. I've focused on the discipline, which is enabled by a toolset. In this way, the only real change from Photoshop to any other tool was the platform. In my Introduction to Forensic Multimedia Analysis courses, the tool is incidental to the delivery of knowledge around the standards, practices, laws, and procedures. Sure, certain platforms make the job easier. In teaching a toolset, like Photoshop or FIVE, I always frame the curriculum around standards compliance and best practices - and present the tool-specific portions after the foundation has been set.

I've written about the new freedom that I'm enjoying now that Amped have retreated to their mail box in Brooklyn. The plethora of classes now on my calendar are not "new." The curriculum development was done years ago. I've wanted to offer these for quite some time to the general market. I have presented them to select US federal service staff on several occasions - causing problems in the process. You see, I was not "allowed" to do so by Amped. Their thinking was simple, I was the only one offering these classes. It would cause a business problem for them if their customers in Russia, for example, wanted to take the courses given that I, as a US citizen, would not be able to deliver them. Amped, having been founded in the epicenter of Italian Marxism, wanted a standard offering at a standard price. Amped, being an Italian company, is free to offer it's products and services to agencies in countries prohibited to US based businesses, like Russia.

If you want to pay a premium to receive your information from a person with an "ampedsoftware.com" email address, please do so with my blessing. If you want real-world guidance, training to competency, and an emphasis on science and standards, designed and delivered by someone who has been there and is still doing that, I'd love to see you in one of my classes. You can find our training calendar here.

Until then, have a great day my friends.

Tuesday, July 9, 2019

FRE Rule 26 asks, where are your notes?

Remember your math teacher demanding that you "show your work." Forensic science is no different. We want to see your process, your thoughts, your notes ...

"The law, however, expects expert witnesses to write reports for the court, outlining their opinions on the matters about which they are testifying. In the eyes of the law, the expert's testimony is allowed by the court only to the extent that the expert adequately documents the basis for it in a report and makes it available to the opposing party before the trial." - from A Guide to Forensic Testimony: The Art and Practice of Presenting Testimony As An Expert Technical Witness (link).

The key to understanding how the legal process requires you to document your opinions lies in FRE Rule 26.

(2) Disclosure of Expert Testimony.
In addition to the disclosures required by paragraph (1), a party shall disclose to other parties the identity of any person who may be used at trial to present evidence under Rules 702, 703, or 705 of the Federal Rules of Evidence.

Except as otherwise stipulated or directed by the court, this disclosure shall, with respect to a witness who is retained or specially employed to provide expert testimony in the case or whose duties as an employee of the party regularly involve giving expert testimony, be accompanied by a written report prepared and signed by the witness. The report shall contain a complete statement of all opinions to be expressed and the basis and reasons therefore; the data or other information considered by the witness in forming the opinions; any exhibits to be used as a summary of or support for the opinions; the qualifications of the witness, including a list of all publications authored by the witness within the preceding ten years; the compensation to be paid for the study and testimony; and a listing of any other cases in which the witness has testified as an expert at trial or by deposition within the preceding four years.

These disclosures shall be made at the times and in the sequence directed by the court. In the absence of other directions from the court or stipulation by the parties, the disclosures shall be made at least 90 days before the trial date or the date the case is to be ready for trial or, if the evidence is intended solely to contradict or rebut evidence on the same subject matter identified by another party under paragraph (2)(B), within 30 days after the disclosure made by the other party. The parties shall supplement these disclosures when required under subdivision (e)(1).

You see the emphasis made on turning over everything in discovery. Notes, reports, and your complete CV. In the world of e-discovery ... this also means notes and reports generated by the programs that we use - if any are created.This means the History Log, in Photoshop or Amped FIVE, is more than likely discoverable. It's a record of the steps the software has taken (at your direction) on an image or series of images, video, or ...

Bottom line, take good notes. If you'd like to know more about taking good notes, standards compliance, and other foundational aspects of forensic video analysis, check out our on-line course Introduction to Forensic Multimedia Analysis.

Monday, July 8, 2019

New Course Announcement

Today marks the beginning of a new era in our training business. I'm happy to announce that Introduction to Forensic Multimedia Analysis is now available on-line as micro learning.

What is Introduction to Forensic Multimedia Analysis? What does it cover?

This class serves as the entry point for those analysts involved in forensic image and video analysis, aka Forensic Multimedia Analysis. It features a detailed overview of the technological, legal, and personnel aspects of working with multimedia evidence in a forensic science setting. Graduates will acquire an introductory level education in the knowledge necessary to evaluate and analyze multimedia evidence in a forensic science setting. With an emphasis on standards compliance, students will also learn the best practices of reporting, packaging, delivering, and presenting their findings and evidence in the criminal / civil court context.

The curriculum is presented with strong emphasis on the work flow required to process this unique type of evidence. Beginning with the end in mind, each example is presented and students learn from the standpoint of an eventual testimonial experience.

In focusing on fundamental knowledge, the information in this course is not specific to any analysis platform or software solution.

Yes. That's right. Regardless of the tools you use, this class can help contextualize the work. It fills in the gaps that often happen with tool-specific training courses.

You've learned which buttons to press. You've learned how the tools work. Now learn the standards and the science behind the discipline. This knowledge - being able to answer "why" type questions, is essential on your career path as an analyst.

As an example, from the course's content, do you know the fundamental difference between the HSL and HSB colour spaces? HSL was designed to closely align with the way human vision perceives colour-making attributes. HSB was designed as a way to try to describe colour more intuitively than with RGB numbers. This level of detail will help you understand the tools that you use at a deeper level, as well as help flavour your testimony - proving plain meaning to industry jargon.

In the next few months, we'll be rolling out other courses in our online portal ... starting with Introduction to Image Authentication. Then, we'll bring out our offerings around the Amped and Adobe tools, followed by deep dives into some of the more popular freeware tools, like ffmpeg.

Here's why:

Many who have been given the task to work on this incredibly valuable evidence type live in states that restrict travel. Others work in cities, counties, and states with budget problems. Traveling to training thus becomes a problem. For every dollar spent on training, two or more dollars must be spent on logistics. That formula isn't sustainable.

Often times, analysts in small towns are disadvantaged as they lack the funds to travel and they lack the population density to attract a training company to their location. Additionally, many municipalities have strict rules prohibiting private companies from profiting off of government resources (no more trading seats for space). Some training companies have thresholds of attendance that must be met in order to hold the training. Signing up, only to have the class cancelled a few weeks out, can cause chaos with your training office.

Another issue that our learning portal solves is the disparity between public and private labs in terms of training opportunities. The majority of training classes are held at government / law enforcement facilities and restricted to those types of employees.  Thus, how are those in private practice supposed to train? With us, they can. Everyone can. Remember, there really are no "sides" in science. We all serve the Trier of Fact in interpreting, analyzing, and explaining the issues around this vital type of evidence.

We've worked hard to check all of the boxes in delivering these offerings to you:

  • Affordable
  • Convenient
  • Relevant
  • Feature Packed
Our introductory education courses are priced below $500. This means that approvals will be easier. This means that you can likely justify the expense and use your purchase card to sign up for classes. Also, given the nature of on-line micro learning, this course is priced to be affordable wherever in the world you happen to be.

Yes, we've designed the course with the world in mind. So many courses are designed with a specific country's context informing the development. We've blended relevant standards and procedural information from Australia, Canada, India, Europe, New Zealand, South Africa, the United Kingdon, and the US ... as well as international standards from the ASTM the world wide working groups.

Having a world wide context, and a platform available on all major device types (even iOS and Android), makes this course ... and all of our courses ... incredibly convenient.

As for relevance, we're filling in the gaps and providing context to what you may already know. We're diving deeper than you've likely gone before. We're setting the scientific foundation for the work. Given that forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context, this course presents the system and leads the way to a coherent study of traces to address the questions that come up in your inquiries. It will provide valuable information and knowledge that will help you make sense of the tools and processes used in this complex discipline.

The course is presented as a 16 hour experience - meaning this course would normally fill two full days of instruction. But, with the use of interactive text, you can dive as deep into topics as you desire. If you dive deep on all topics in the course, this course expands to beyond 40 traditional classroom hours of instruction. As a self-directed learner, the choice is yours.

The beauty of micro learning is that you learn at your own pace and on your own schedule. All too often, the "death by Powerpoint" section of class puts students to sleep. In sleeping through the lessons, they miss vital information. Micro learning lets you explore topics in depth, or in small bites, as your schedule and energy level permits. Once enrolled, you have 60 days to complete the course.

I invite you to examine the syllabus for this course. Then, please click over to the web site and sign up. I look forward to seeing you in class.


Tuesday, July 2, 2019

Yep! They Did It Again.

I've been teaching classes on multimedia analysis for a while now. Almost a decade ago, I created a curriculum and began teaching classes for Amped Software, Inc. in the US and elsewhere. One of the things I made sure to focus on was the specific use case for my students. If I was in Georgia, I would focus on Georgia law. Likewise for Ontario Provence or South Africa.

In taking a walk through the program, I would point out the Program Options within FIVE (View>Program Options), and we'd spend the better part of the morning going over each setting and it's implications for the analyst's work.

What took the most time to explain was the Log Files option. By default, Amped SRL's FIVE creates a log file that records every step from the moment you launch the program to the time you shut it down. It has since Build 4376, as I noted in this old post from 2012.

Amped SRL's Marco Fontani explains, in this recent blog post, what a wonderful thing the log files are - in his opinion. It's important to remember that Marco is in Italy, where the rules and laws are quite different from elsewhere in the world.

What is a log file? What's in it? Marco explains:

We open the file and find an impressive amount of information. The file begins with the product info: release number and build date, info about the computer and operating system, info about installed applications that may influence report creation (Microsoft Word in this case). Then, we have a full history log of every move we made while using Amped FIVE: which filter we added, how we configured (and possibly re-configured) it, which filter we deleted, and so on.

Wait. What?

You read that right. In case you missed my 2012 blog post, everything about your operating environment. Every single step. The time each step was performed. Everything. Recorded.

Did you know that FIVE was doing that?

If you have been to one of my classes, you do. If you're a long time reader of this blog, you do. I'm the one that told you to turn off this feature and only activate it under one of two very specific situations.

  1. If you were having an issues with FIVE and you were specifically told by an Amped Software, Inc, employee to activate it in order to step through the problem.
  2. Outside of troubleshooting - if you have specific permission from your chain of command to generate / create this type of potentially exculpatory information for your case work
If you received a discovery request for any / all files related to the case in which you were a part, and you didn't disclose this information (that you didn't know existed). Once created, they have to be saved with the case. Please don't delete potentially exculpatory or other case related info unless specifically directed to by your chain of command (or by statute). This isn't legal advice as such, it's just my experience as an analyst in government service who also served as a Union Rep & Shop Steward.

The other little Jack-in-the-Box to Marco's post, the one he hints at but doesn't say explicitly, is that the log is a record of everything from Open of FIVE to Close. Why would that be important?

If you've been in my classes, you'll remember me saying (rather regularly) to shut down FIVE after processing your case and restart it to begin a new case. Yes. I told you about the log files. But, just in case you forgot later, and FIVE resets itself or you move to a different computer, I wanted to get you in the habit of refreshing FIVE between cases. Why?

Because if the log is a record of everything done, every step accepted or rejected, from Open of FIVE to Close, AND don't shut FIVE down between cases, you've just mixed cases on the same log file.

Yes, you did that and you didn't know it.  

Now, each case gets the record of what you did on the other case(s). Did you just work 30 cases without closing FIVE. Surprise, all 30 cases are on the same log file. All 30 cases' attorneys get to know what you did or didn't do on the other cases. Can you imagine the cross examination nightmare?

In case you're wondering what this might look like, here's a peek from 2009 - FRE Rule 26 and the History Log.

Before you say it, I'm not suggesting you hide anything. What I am saying is that you should, you must, seek your chain of command's counsel before activating that feature. When I did, I was told very frankly to shut it off. You see, at the LAPD, someone of my rank/pay grade did not have permission to create a new "form," which is essentially what the log file is.

Friday, June 28, 2019

Junk Science?

Great news from Texas has brought out a bunch of people with conflicting agendas. The great news? The George Powell case has moved to retrial after the Texas Court of Criminal Appeals vacated the conviction and ordered a new trial.

Whilst the media buzzes about the evidence used in the original trial, and throws "junk science" in the same sentence as "forensic video analysis," most in the media commit journalistic misconduct in their reporting of the case and the circumstances around the re-trial.

Take this article (link) for instance, "George Powell has maintained his innocence the entire time he's been behind bars. Now, he will have a chance to prove it."

WRONG.

In vacating the conviction, George Powell's status has returned to Innocent as the "proof" of his guilt has been vacated or struck down. Powell, under US law, is innocent until proven guilty in a court of law. As a presumed innocent man, he should be afforded the opportunity to make bail and to prepare for his defense.

When the trial begins, it's for the prosecution to offer the evidence that links Powell to the crime, if such evidence exists. It perverts the course of justice to require Powell to prove his innocence. Why? Not only is it the way our system works, innocent until proven guilty, but it's impossible to prove a negative.

Thus, when the trial begins, it's not "forensic science" that will be on trial. It's not "forensic video analysis" that will be on trial. What will be on trial is the evidence, offered up as proof of some condition. Then, it's up to the Trier of Fact to evaluate that evidence.

Forensic science and forensic video analysis are scientific, when performed scientifically by educated, trained, and proficient practitioners utilizing valid tools and an appropriate and valid methodology.

As regards Photogrammetry specifically, a measurement shouldn't be used to "identify" an object or a subject. Rare is the case that there is nominal resolution in the evidence item sufficient to measure with an output range that is so very precise. If the measurement's range of values in this case, properly employed, is between 5'6" and 5'10", as an example, then more than 65% of the population of the area will fit within that range. It's hardly an identifying characteristic. It does, however, help prioritize tips as well as to help exclude from the enquiry those subjects more than a standard deviation away from that range. This is why it's important to engage in valid methods when performing an analysis.

Tuesday, June 25, 2019

So you want to dive deep?

During last week's class, and at the request of several who wrote ahead to make sure that we were going to "dive deep," we dove deep into science. Here's an example of how deep the discussions often go:

In a modern chemistry textbook (link), students read the following:

"Matter comes in many forms. The fundamental building blocks of matter are atoms and molecules. These particles make up elements and compounds. An atom is the smallest unit of an element that maintains the chemical identity of that element. An element is a pure substance that cannot be broken down into simpler, stable substances and is made of one type of atom."

Sound good, so far? No, actually. There's so much missing.

For most people, this seems simple. The statements are matter-of-fact about how material objects exist.  We read them and assume that this has been seen and proven and when we look at a piece of wood or a drop of water, we tell ourselves that beyond the reach of our eyes, there is actually no “wood” or “water” but simply atoms arranged in different ways to cause different substances to appear to us, like an Impressionist painting composed of dots of paint.

If we move back, however, to the early 1900s, students were taught the following in The Elements of Chemistry: Inorganic and Organic. Norton, S.A. 1884 (link):

"We know nothing of the manner in which the ultimate particles of matter are arranged together: we believe that they are arranged in accordance with certain theories which we shall now proceed to develop.

All masses of matter may be subdivided into very small particles; but it is probable that there is a limit to this subdivision, and that all bodies are made up of particles so infinitesimally small that they are inappreciable to our senses. By the terms of this theory,

A molecule is the smallest particle of matter capable of existing in the free state:

An atom is the smallest particle of matter that is capable of entering into or existing in a state of chemical combination."

Notice how carefully the old teachers distinguished facts from theories.  The textbooks tells students plainly that scientists believe in theories and that teaching about atoms and molecules is a matter of faith not facts.

In this same textbook, Norton warns of errors that people are prone to when they speak of the science and its facts.  We read:

"The facts of chemistry are established by experiment, and are capable of being reproduced. They find a practical application in the arts, which is altogether independent of any explanation that may be made of them. When, however, we attempt to reason upon these facts, to classify them, to interpret them, we at once begin to form theories. A theory which renders a reasonable explanation of a great number of facts is useful (1) because it enables us to group them into a system, and (2) because it often leads to new experiments and to the discovery of other facts.

We are liable to three errors: (1) we may assume that to be a fact which has no existence; or (2) we may sometimes mistake a phenomenon, so as to imagine that to be a cause which is only an effect of some unknown cause; or, finally, (3) we may become so accustomed to the language of theory as to mistake its definitions for facts."

Can you  not see how different the mind of the student formed by this older textbook is than the modern textbook which commits the third error learned of?  Teaching of atoms and molecules, in modern textbooks, is no longer couched in terms of  theory and belief.  It is presented as immutable fact.  What is fact is that there is no more proof that atoms actually exist today than there was in 1900. The language of theory has simply been abandoned by modern scientists.

Swap out the word "chemistry" for "video analysis" or your specific forensic science discipline, Norton's caution still makes for great advice - from 1884.

This simple discussion, generic to science, has profound implications in the forensic sciences. I speak often of what we "know" vs. what we can "prove," of the null hypothesis, and of conducting valid experiments. Remember the NAS Report's conclusion in 2009 - forensic science sucks at science. Well, how does forensic science remedy the situation - by engaging in actual science. That all starts with understanding how science is conducted.

As relates to measurements, comparisons, vehicle determinations, and etc., we start with the basics. To get to a vehicle determination, we must first have a nominal resolution sufficient to address the task. Same with photogrammetry - we need sufficient nominal resolution of the item or object of interest so that the measure will (at least) exceed the error. We also must understand that a measurement can't be used to identify something. The measure only contributes to the classification of the item or the object.

So, if you want to dive deep - really deep - our classes are open to all who have an enquiring mind. Even our introductory courses move beyond simple button pushing to explore the depth of the discipline.

Friday, June 21, 2019

Practical Field Photogrammetry

Another successful Photogrammetry class is in the books. I'd like to share a few thoughts on the experience, as well as an example we worked on in one of the field exercises.

The old adage, "a poor workman blames his tools," is only part right. If you don't have the right tools to begin with, you'll be up against it. In the justice system, analysts are often stuck with what they've been provided - by the lowest bidder. This is something I realized years ago in that crime factory of a city that I used to work in. I needed a kit that was economical, rugged, portable, and easy to work with. It needed to be economical because the expense of kitting myself out was coming out of my own pocket.

As regards measurement exercises, I've seen many height charts / resolution charts on offer at trade shows and on-line. The ones in my price range were often printed on ink jet and when mounted, mounted on cardboard with a rigid frame. They were bulky and awkward. They maybe would last a month of regular use.

Having a background in art, working as a commercial artist and having run art departments in my youth, I decided to design and manufacture my own resolution chart.


Beginning with the end in mind, I decided to not go with the standard checkerboard style. I didn't want the black boxes to touch each other. At a distance, and given the massive compression on most surveillance systems, I wanted to avoid the bleed-over that often happens with the checkerboard designs. Take this from the Amped FIVE sample folder, as an example of the bleed-over.


This image is a close up, and in the camera is less than 12 feet from the chart. Still, you can see the problems of trying to establish the corners for an "eyeball" nominal resolution calculation.

The types of enquiries we often face deal with questions of license / registration plates at distances of 50'-250'. Take this example from class.


The camera is mounted on a light pole about 100' from the target, elevated about 12' from the road. I've processed it for display here in FIVE. The system is on the higher end of the generic black-box Chinese DVR spectrum and is well maintained. The owner of the system is a friend and the strip mall isn't that popular, so it's a nice place to have a field day.

My resolution chart is commercially printed on a PVC board. It's durable, water proof, and can stand propped up against the back of the car. Thus, I don't need to carry a stand or easel. It fits nicely behind the driver's seat of my truck.


Cropped in on the back of the car and the chart, this is the view at 100'. The left-most blocks are 2", the middle squares are 1", and the right-most squares are 1/2". There's very little bleed or compression artifacts around the 2" squares.


There are two stickers on the rear window. What's the diameter of the top sticker? I've got a good look at the 2" squares, so I can use those as my references. I can also use them to calculate the nominal resolution. If I use the Ruler in FIVE to measure the pixel length of the 2" squares, I get 4px.  Thus, the nominal resolution around the rear of the car is 2px/in.


Moving on to measurements of the items of interest on the vehicle, accurately selecting the X and Y axis in FIVE takes just a minute. The results are displayed in seconds. Things like stickers help to "individualize" the car and differentiate it from the other Honda Fits in the general area. This one has a 6" diameter round sticker on the center left of the rear hatch's window, above another sticker.


We can check our work by measuring an object of a known length.


US license plates are 12" long. This car appears to have a license plate frame. We measured from the outside of the frame on both sides. The results are again displayed. Assuming a bit of error (Measure 2D in FIVE does not compute error as 2D measure is simple planar geometry), at 100' from the camera and with compression, that's not a bad result - 1/2" frame width on both sides is not out of the realm of possibility.

What makes this possible? An easy to use resolution chart.

This is a simple example. But the technique is the same for your basic reverse projection photogrammetry experiment. Make a recording of the scene with the chart placed appropriately, then overlay the two recordings (that's the reverse projection part) in order to facilitate the measurement (that's the photogrammetry part). In this case, everything is roughly on the same plane, so I used FIVE's measure 2D. FIVE has 1D, 2D, and 3D measurement capabilities.

The last part of this exercise was the validation of the results. Validating results, in each and every case, is a must. There is no research out there that can be utilized that "globally" validates reverse projection photogrammetry. Every case is unique - different camera, different lens, different DVR, etc. Thus, you validate. The few text books out there that mention RP advise validating each case. In the SWGDE doc on photogrammetry (link), they advise the analyst to calculate / determine "... the accuracy and precision of this result." That's just another way of saying, "validate your results."

Keep in mind that if you're using a mixed-methods approach (laser scan point cloud + still CCTV frame), that method has never been validated outside of some analyst's casework. Remember, getting evidence and testimony accepted in one venue is not validation. So please, do your validations.

There's one more Photogrammetry class scheduled for 2019. You can check out the calendar of classes on offer here. This course tends to fill fast as there are no "non-vendor specific" courses on the subject of photogrammetry available in North America. It's a very intense week as we dive deep into the science and practice. It's a very tiring class to teach. It's a lot of information to transfer in a week's time. But, I enjoy it.

When students take this course, they've usually been through the LEVA levels and the various tool-specific training sessions. They're looking for more training, or topic-specific training. They're usually well kitted-out for the basic stuff. Thus, what do you get for the person who has everything? A squeeze ball and a few branded pens? No. Students get the digital printing file of the height chart and instructions on contacting the commercial printer we use. The cost of printing and shipping is usually less than the cost you'll find in the forensic magazines or at trade shows for their charts. If you can't make the class and would like the file, a small stipend is all it takes. If you'd like me to arrange the printing and have it shipped to you, the small stipend + the printing / shipping costs is still reasonable (I don't resell the printing/shipping). Either way, just send me a note and we'll get it set up for you.

Until then, have a great summer ...

Monday, June 17, 2019

Forensic Photogrammetric Analysis

This week, I'm happy to be teaching one of my favourite subjects, Photogrammetry.

Unlike the many tool-specific courses offered by the many vendors in the laser scanning space, the Forensic Photogrammetric Analysis course that I've been teaching for a while now focusses on the science of measurement as well as best practices that will help arrive at valid conclusions.

Here's a bit of a sample of what we're on about this week...

A few months ago, I wrote about calculating Nominal Resolution during the content triage stage of the image / video processing workflow. That critical process depends upon an accurate measurement of the distance from the camera to the object of interest. Who's responsibility is recording this measurement? Did you collect the video from the crime scene / witness location? If you did, did you roll out your tape measure or use your roller measuring wheel? If not, why not?

If not, might it be due to a lack of knowledge about the importance of calculating the nominal resolution? Perhaps you simply don't have the right kit.

You can buy your tools from a dedicated forensic science / crime scene vendor like Sirchie. Or, you can head over to your local home improvement store. At Home Depot, you can get a 300' measuring tape for under $40.

This Lufkin measuring tape is currently $38. A similar measuring tape from Sirchie is almost $80.

The same goes for a measuring wheel.


The Lufkin measuring wheel at Home Depot is less than $35. A similar model at Sirchie is over $70.

The point?

It doesn't have to break the bank to perform measurements at scenes that will later inform measurements within your images and videos.

If you'd like to know more, or if you'd like to sign up for our next class this fall, head over to our training page and see what's on offer. I hope to see you in class soon.

Tuesday, June 11, 2019

Retrieval / Seizure of Electronic Evidence from Crime Scenes

Twelve years ago, a few multimedia analysts got together via emails and phone calls to create a training class around an emerging topic - what to do with the video contained within these black boxes that folks were encountering at crime scenes.

Several of us had "figured it out." We figured out what worked in our own contexts. What worked for me in Los Angeles was slightly different than what worked in New York, or Virginia, or Texas.

We hashed out a loose curriculum for what was eventually delivered at the 2007 NaTIA conference in Pittsburgh. In that course, we presented a hands-on overview of the technology, the legal aspects, and ideas for procedures that could easily be implemented by state / local agencies.

This experience lead to the creation of the red flip book, aka Best Practices for the Retrieval of Video Evidence from Digital CCTV Systems. I'm quite proud of what we all accomplished in the production of that handy guide, which is still available from the CCTSO.

From there, the group was brought together again to create the curriculum for what would become the RECVR class at FLETC.

From that point, several of us went on to create separate versions of the essential materials, delivering it across the county (and the world). I've had the privilege of presenting this topic across the US and Canada, as well as in Pretoria, South Africa to the South African Police Service.

Whilst most classes need to choose a platform from which to work, and you'll need tools to perform essential work in restoration, clarification, and analysis, my curriculum has always attempted to offer a "non-platform specific" option to learners. I've presented as many tools as possible, many of them free / open source. I've focussed on policy, procedure, and workflow. I've split the topics around retrieval from the topics around seizure. I think the curriculum is rather comprehensive, yet accessible to learners of all types (novices - advanced analysts).

During my years with Amped, there was a lot of internal tension around those classes that I had previously designed and presented being offered by an Amped employee. Thus, they weren't. There was also a bit of angst about the way in which I presented the training around the Amped tools. I don't like simply focusing on buttons and functions. I try to contextualize the tools, and thus present the training in a way that builds competency.

So, with Amped's retreat, I'm happy to say that my full slate of classes is back on offer. You can find them on all listed by clicking hereRetrieval / Seizure of Electronic Evidence from Crime Scenes is back and better than ever. It doesn't really compete directly with the other classes out there. On the contrary, it's entirely complimentary.

You see, every instructor and curriculum developer has their own point of view and experiences. Each brings all of that to their courses. I'm no different in that respect. I don't usually toot my own horn, but this is different. My story helps frame the creation of the classes. Here it is:

  • I'm a Certified Audio/Video Forensic Analyst (AVFA). The process of investigating the three bodies that offer certification, honestly apprising the costs/benefits of each, deciding on a path, and carrying it out to completion has been an over-a-decade journey.
  • I'm the author of the best-selling book Forensic Photoshop (available on Amazon.com). Getting a book published isn't as easy as it seems. Ten years ago, it was a daunting task. Putting yourself out there for others to critique is an amazingly stressful ordeal. I'm glad that I did it.
  • I'm a co-author of Best Practices for the Retrieval of Video Evidence from Digital CCTV Systems (red flip book). Collaborating with an amazing group of international all-stars in the field, listening to their diverse views, creating, compromising, standing firm, and finally seeing one's work in printed form was an awesome experience.
  • When I retired from the LAPD, I had spent a third of my life in police service. Based on an 8 hour work day, and the amount of actual hours I had worked, I had worked 33 years worth of 8 hour days during that time. During the last half of my career,  I hardly saw my family and friends, my health suffered, and I was heading towards burn-out. Given the amount of crime in LA, and the relatively small size of my old group, a lot of the way in which I work and structure my courses comes from the "factory of crime" and the need to work efficiently in the face of constant requests for service. Also, my focus on human factors, taking care of one's self, and doing one's best to eliminate bias comes from this experience.
  • I currently serve the community on the Organization of Scientific Area Committees on Forensic Science (OSAC), Video / Image Technology and Analysis (VITAL), as the Video Task Group Chair. Last year, we moved the DCCTV Guide to an ASTM standard. This year, we're about ready to move the many SWG docs around forensic video analysis to a common ASTM standard.
All of this finds its way into my classes. I'm not just a talking head. I'm not just reading a script. I'm giving my honest assessment of the topic, and a helpful set of instructions to get the learner to mastery.

If you're ready to move beyond "button pushing," head over to Apex Partners to check out what's on offer. Sign up for a class or two. I hope to see you soon.

Friday, June 7, 2019

Forensic Science or Investigative Support

In 2015, then Deputy Attorney General Sally Quillian Yates announced that the Justice Department will, within the next five years, require DOJ-run forensic labs to obtain and maintain accreditation and require all department prosecutors to use accredited labs to process forensic evidence when practicable. Additionally, the department has decided to use its grant funding mechanisms to encourage other labs around the country to pursue accreditation.

If all that you read in the announcement was the headline and the first few paragraphs, you'd be left with the impression that the entirety of forensic science functions under the DOJ's control would be a part of this new initiative. You'd believe that ... and you'd be wrong.

In the fifth paragraph of the announcement was this, "The new policy does not apply to digital forensic labs. Instead, the Deputy Attorney General has asked the NCFS to develop separate recommendations on accrediting of labs that conduct digital forensic work, given the difference in the practices of forensic analysis of digital evidence."

This requires a question be asked, is the analysis of digital evidence a forensic science or investigative support function?

The FBI's FAVIAU is part of the Operations Technology Division. From their web site, "The world-class capabilities developed and deployed by the Operational Technology Division (OTD) have been instrumental in averting a terrorist plot, identifying adversaries involved in espionage activities, and helping to convict a child pornography subject. And these are just a few examples of where OTD has provided technical support in developing and deploying a capability in an FBI investigation or national security operation. OTD’s capabilities can be categorized into seven areas, all of which are used across the Bureau’s intelligence, national security, and law enforcement operations."

It seems that the FBI considers it's digital evidence labs to primarily serve an investigative support function, even though FAVIAU and the many RCFLs are fully functional and accredited forensic science laboratories. It's also worth noting that the digital evidence functions aren't part of the FBI's actual laboratory.

Similarly, the LAPD (my former employer), does not house it's digital evidence functions within its crime laboratory. My old unit (the Electronics Unit), is a part of the Technical Investigation Division, which is a part of the Detective Bureau.

"The Technical Laboratory is comprised of four specialized units that provide support services to investigative personnel in the Department - the Latent Print Unit, the Photographic Unit, the Polygraph Unit, and the Electronics Unit. Most Technical Laboratory personnel operate out of Piper Technical Center. Some field services operate out of the Van Nuys Community Police Station."

My old unit handles audio, video, and mobile device forensics - very similar to the coverage provided by the FBI's FAVIAU. Like the FBI, computers ("digital evidence") are handled by a separate unit. At the FBI, there's the CART team. At the LAPD, there's the Computer Crimes Unit. The LAPD also has several siloed digital evidence "labs" within sensitive / specialized units. I suspect the FBI does as well.

I know from my own career at the LAPD that it's entirely possible to "serve two masters," investigative support and forensic science. It's possible to work fast (investigative) and accurately (science). The modern toolset has helped tremendously.

But, within this seeming split, it's important to note that the other forensic science disciplines also serve this dual function. DNA results are used in investigations to rule in/out persons of interest. Same for latent print and firearms results.

So why the separation?

I think the separation has a lot to do with compliance / accreditation issues. The perception is out there that accrediting digital laboratories under ISO/IEC 17025 is complex and burdensome, and because accreditation groups like ANAB allow agencies to determine which aspects of its testing, calibration, and/or inspection services to accredit, that it's much easier to exclude digital forensic labs than to include them. In this case, perception is not reality. Accreditation of digital evidence labs is actually pretty straightforward.

If you'd like to know more, or if you're exploring accreditation for your lab, let me know. There's a clear path forward that many have taken. You're not alone. I can help.

Wednesday, June 5, 2019

I'm AVFA Certified

It's one thing to take a certification in some random Prometric testing center. It's quite another to get your certification package in the mail. Yay!


It was a pretty thick package. Certificate, ID card, membership info, program info, forms, and etc. Everything arrived in a week. That's pretty fast, considering the distance.


One of the interesting items was a set of forms for submitting my continuing education information annually. They do want to make sure that you're continuing to educate / train yourself within your chosen discipline. It's interesting in that they want the information annually, not 4 years later (oh, crap! I don't have enough CE's. What do I do now?!...)

In all, getting certified was a refreshingly pleasant experience. No drama. No stress. Very professional.

Now, it's back to work ... If you're interested, seats are still available at next month's Introduction to Forensic Multimedia Analysis with Amped FIVE course at our office in Henderson, NV.

Friday, May 31, 2019

Is it practical?

Continuing the discussion of certification in general, the NAS Report notes the following about certification programs, "The American Bar Association has recommended that certification standards be required of examiners, including “demanding written examinations, proficiency testing, continuing education, recertification procedures, an ethical code, and effective disciplinary procedures.” Let's break these down.

Demanding written exams. What does this mean? Are "written exams" composed of essay questions where one explains topics in depth? Are "written exams" objective tests, like multiple choice exams? What makes an exam "demanding?"

Could an examination body survey the entirety of my written work, books, papers, speeches, this blog, and so on? Could they judge my proficiency based on my written work? I'm not sure. But, pulling from my academic life, it is entirely possible for someone to be proficient in their craft, but be non-verbal. How do you conduct a "demanding written exam" on a non-verbal person? What about differences of language? Do we insist upon all tests being in US English? How do we handle the issue of testing vulnerable populations? Given my extensive work in the autistic community, both academically and professionally, these are relevant questions.

Proficiency testing. Does this belong to an independent certification program or within a vendor's training program? For example, who is a better judge of my proficiency with my chosen tools, Amped Software, Inc. (vendor) or the IAI (third party)? What does the IAI know about FIVE and Authenticate? Nothing, actually. They're still focussed on Photoshop. How do I know this? Their reading list has George Reis' Photoshop book as one of the sources for the test. Whilst still relevant, and quite good, it's a 10 year old treatise on George's favorite PS techniques in no particular order. Given that I no longer use Photoshop in my analytical workflow, how is a test on Photoshop helpful in assessing my proficiency as an examiner?

Perhaps the "requirement" could be met with a review of an analyst's case documentation. Submit a few reports, have the panel examine them in light of the ASTM and SWGDE standards / guidelines.

Continuing education. There comes a point in time, when you've been doing this for a while, that you run out of classes to take. Some, that are available, are cost prohibitive. That's a large part of why I've moved a lot of the classes that I teach to an LMS as micro-learning. As an example, if you can't get in to your local college to take a statistics course, you can take Statistics for Forensic Analysts on-line at an affordable price. Same for redaction, Redaction for Standards Compliance is offered on-line featuring a variety of software platforms.

I've been around the US and Canada quite a bit. I realize that traveling to training is not always an option. I think the internet offers some amazing options to level the playing field as regards continuing education. The question becomes, will your chosen certification program accept your completion certificate from an on-line course or provider?

Recertification procedures. I think the important part here is not only that the individual circle back and re-certify, but that the certification program continually monitors the industry and keeps the certification program up to date. Four years from now, when my AVFA certification is due for renewal, I should not be presented with the same test instrument. Laws and technology change. So too should the certification program. Again, I love George Reis to death, but is his book still relevant in a modern multimedia workflow? Perhaps. But are there other, more appropriate texts? Yes. Absolutely. What about listing Dr. John Russ' Image Processing Cookbook as well, given the preference for Photoshop? Or his Image Processing Handbook, for a more academic look at image processing? If price is an issue, they could choose Fundamentals of Digital Image Processing by Dr. Anil Jain. It's a text book that can be bought used for under $10.

Ethical code. Here's LEVA's Code of Ethics. This post is an example of article 5 in LEVA's COE, offering constructive criticism. Interestingly, article 9 is what prevented me from ever being LEVA certified. Article 9 states, "An Analyst shall maintain the confidentiality of evidence received and work undertaken in accordance with any such request by the submitting agency." It was the policy of the Los Angeles Police Dept. that case work remain confidential and no evidence leave the secure facility without permission. When I sought permission to begin the process of certification, it was flatly denied. Given the LEVA certification methodology, if you can't present one of your cases, you can't get certified. Thus, in honoring their COE, I was prevented from ever being certified whilst in police service. Now, in retirement and in private practice, LEVA discriminates against me for not being a current public service employee by charging confiscatory prices for their offerings.

For the IAI's COE, click here. It begins on pg 79 of the linked document.

Effective disciplinary procedures. One of the things that we do is review opposing counsel's analysis. The examiner on the other side of any case should expect a thorough review of not only their work but their credentials as submitted through the discovery process. I have lost track of the amount of times I have encountered embellishments and fabrication of credentials by analysts, some of which have been certified by LEVA or the IAI. Everything you write in your report should be factual. Everything you write about yourself should be verifiable. As an example of this, I note as a matter of fact that I am retired from police service, having served just shy of 15 years with the Los Angeles Police Dept. under civil service classification 3687. This is a verifiable statement. I can provide the job description for civil service classification 3687, or you can click on the link in the previous sentence. I can provide my retiree ID. I could even print out an account statement from the City's retirement plan noting how many years/months I have of retirement credits (an indication of years/months worked). But, if I simply quit my job, it is a slightly different process than retiring. Thus, I wouldn't say that I had retired, but that I had quit / separated / moved on - or some other term that would accurately describe the end of my time in police service. Same goes for my education. No embellishment, just a simple presentation of the facts.

Back to ethics, there have been a few times that I reported my findings to a certification body. To the best of my knowledge, nothing was ever done with the information that I provided. This is the problem with most professional organizations. We're all friends. No one wants to judge their friends. No waves. No trouble. Let it slide. If there are no consequences, there's no effective discipline.

As these all relate to the certification experience that I just undertook, the AVFA includes all elements except the "proficiency test." I was not given an item of evidence and asked to examine it vs. a question under enquiry. Could my body of work stand in a proficiency test's stead? How much have I written? How many courses of instruction have I designed? How many have I trained to competency? How many cases have I worked? After all, it's this body of work that prepared me to take and pass the exam.

Getting back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." My education, training, and experience prepared me to take and pass the examination that was prepared by peers, thus demonstrating to the certificate program a minimum level of competence. As to the peers who prepared the exam, I may not know a single one of them, but having a list of names with contact information is a great start.

Thursday, May 30, 2019

Taking the AVFA Certification Exam

I have to say that signing up and taking the AVFA certification exam was a rather pleasant experience. Yes, that seems a bit odd to say. But, it's true. It should be true that an organization treats you like an adult, respectfully walking you through the process of engaging their services.

I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.


You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.

I can't emphasize this enough, I paid $120 in fees to take this certification exam.

If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.

For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.

At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.

Back to the experience.

I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.

The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.

The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.

In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certificate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.

Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.

Wednesday, May 29, 2019

The AVFA Certification

On November 22, 2005, the Science, State, Justice, Commerce, and Related Agencies Appropriations Act of 2006 became law. Under the terms of the statute, Congress authorized “the National Academy of Sciences to conduct a study on forensic science, as described in the Senate report."

In 2009, the National Academies Press published Strengthening Forensic Science in the United States: A Path Forward (aka, the NAS Report). The report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.

On page 208 of the NAS Report, the authors address the issue of certification.

"The certification of individuals complements the accreditation of laboratories for a total quality assurance program. In other realms of science and technology, professionals, including nurses, physicians, professional engineers, and some laboratorians, typically must be certified before they can practice. The same should be true for forensic scientists who practice and testify. Although the accreditation process primarily addresses the management system, technical methods, and quality of the work of a laboratory (which includes the education and training of staff), certification is a process specifically designed to ensure the competency of the individual examiner."

The NAS Report goes on to note, "In addition to improving quality, certification programs can enhance the credibility of certificate holders."

As I noted last week, there are three certifications available in the discipline of digital / multimedia analysis. I would like to correct that here first, leaving the original post unedited.

Of the domains of what is traditionally considered Image Analysis / Video Analysis, there is no specific component within the LEVA / IAI that addresses the testing of a practitioner's competency in authentication. Photogrammetry is sometimes covered at their conventions, but there again, it's not a specified component of the certification programs.

I can say this with certainty about the LEVA certification. One may stand for certification upon successfully completing the LEVA Levels courses, the Courtroom Prep class, and some electives. For the exam, the analyst chooses a case that they've worked and presents it to the certification board. Thus, what is covered in that "exam" is unique to the analyst - varying from analyst to analyst. As a certificate program, it seems quite out of alignment with ASTM E2659 – 09, Standard Practice for Certificate Programs.

As to the domains under examination, there is no list available from either group. The IAI has a reading list that consists of a few older texts as well as some SWGDE guidelines. It's assumed that the practitioner will gain proficiency in these areas via vendor training, attending workshops at conferences, and other training opportunities.

What's missing: audio.

In Tx vs Oliver, the LEVA CFVA's testimony notes the importance of the audio track of the body-worn video recording, and how it helped in the analysts work. But, the CFVA exam does not specifically test for audio competency. I'm certainly not knocking the analyst. Just noting that the CFVA exam doesn't specifically test for competency in audio analysis and the analyst noted how important the audio track was in guiding his work and his conclusions.

Yes, forensic multimedia analysis includes everything in the multimedia container - audio, video, images, metadata, error, and junk. The CFVA - video analyst - seems to just cover the video portion of that list. Not necessarily a bad thing, but it's important to know what it does and doesn't cover.

Let's say you have an "image" from an iPhone - a Live Photo. Live Photos contain sound (link). Does your chosen software platform recognize that a Live Photo is essentially a video file? Does your chosen software platform have the ability to play the audio? What about "enhancing" the audio? Do you know how to enhance audio?

Here's where the AVFA certification has a leg up. Whilst the "competency requirements" aren't all that well written, and the format of the test (as noted on their site) does not match the language in the comps document, the analyst will definitely have an idea as to what will be on the test.

Yes, the AVFA exam includes audio. It's the first A in the acronym. The AVFA exam includes Photogrammetry and Authentication. There doesn't appear to be anything related to comparisons or content analysis, but those domains can sometimes be blended into the others, or generalized to "analysis."

Back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." It seems that the AVFA meets that definition.

I'm going to take the test soon. I'll let you know how that goes as well as comment on the whole process - from sign-up to notification of results and beyond.

Monday, May 27, 2019

When the data supports no conclusion, just say so

I can't quantify the amount of requests I've received over the years where investigators have asked me to resolve a few pixels or blocks into a license plate or other identifying item. If, at the Content Triage step, nominal resolution isn't sufficient to answer the question, I say so.

When the data supports no conclusion, I try to quantify why. I usually note the nominal resolution and / or the particular defect that may be getting in the way - blur, obstruction, etc. What I don't do is equivocate. Quite the opposite, I try to be very specific as to why the question can't be answered. In this way, there is no ambiguity in my conclusion(s).

Additionally, whomever is responsible for the source of the evidence will have insight as to potential improvements to the situation. For example, if the question is "what is license plate," and the camera is positioned to monitor a parking lot, then the person / company responsible for the security infrastructure can be alerted to the potential need for additional coverage in the area of interest. This could mean additional cameras, or a change in lensing, or ...

Why this topic today?

I received a link to this article in my inbox. "Expert says Merritt truck ‘cannot be excluded’ as vehicle on McStay neighborhood video." Really? "Cannot be excluded?" What does, "cannot be excluded" mean?

At issue is a 2000 Chevy work truck, similar to the one below. Chevy makes some of the most popular trucks in the US, second only to Ford. This case takes place in Southern California, home to more than 20m people from Kern to San Diego counties.


"Cannot be excluded" is not a conclusion, it's an equivocation. Search CarFax.com for used Chevy Silverado 3500 HD trucks for sale within a 100 mile radius of Fallbrook, CA (92028). I did. I found  49 trucks for sale on that site. 49 trucks that "cannot be excluded" ... AutoTrader listed 277 trucks for sale. 277 more trucks that "cannot be excluded" ...

All of this requires us to ask a question, is the goal of a comparative analysis to "exclude" or to "include?" How would you know if you've never taken a course in comparative analysis? Perhaps we can start with the SWGDE Best Practices for Photographic Comparison for All Disciplines. Version: 1.1 (July 18, 2017) (link):

Class Characteristic – A feature of an object that is common to a group of objects.
Individualizing Characteristic – A feature of an object that contributes to differentiating that object from others of its class.

5.2 Examine the photographs to determine if they are sufficient quality to complete an examination, and if the quality will have an effect on the degree to which an examination can be completed. Specific disciplines should define quality criteria, when possible, and how a failure to meet the specified quality criterion will impact results. (This may apply to a portion of the image, or the image as a whole.)
5.2.1 If the specified quality criteria are not met, determine if it is possible to obtain additional images. If the specified quality criteria are not met, and additional images cannot be obtained, this may preclude the examiner from conducting an examination, or the results of the examination may be limited.
5.3 Enhance images as necessary. Refer to ASTM Guide E2825 for Forensic Digital Image Processing.

These steps are the essence of the Content Triage step in the workflow - do I have enough nominal resolution to continue processing and reach a conclusion?

But, there is more to this process than just a comparison of a "known" and an "unknown." How does one go from "unknown" to a "known" for a comparison? How do you "know" what is "known?" First, you must attempt a Vehicle Make / Model Determination of the vehicle in the CCTV footage.

For a Vehicle Make / Model Determination, the SWGDE Vehicle Make/Model Comparison Form. Version: 1.0 (July 11, 2018) (source) is quite helpful.

How many features are shared between model years in a specific manufacturer's product line? Class Characteristics can help get you to "truck," then to "work truck" (presence of exterior cargo containers not typically present in a basic pickup truck), then to "make" based on shapes and positions of features of the items found in the Comparison form. The form can be used to document your findings.

You may get to Make, but getting to Model in low resolution images and video can be frustrating. What's the difference between a Chevy Silverado 1500, 1500LD, 2500HD, 3500HD? There are more than 10 trim variations of the 1500 series alone. What's the difference between a 2500HD and a 3500HD?

After you've documented your process of going from "object" to "work truck" to a specific model of work truck, how do you move beyond class, to make, to model, to year, to a specific truck? Remember, an Individualizing Characteristic is a feature of an object that contributes to differentiating that object from others of its class. Before you say "headlight spread pattern," please know that there is no valid research supporting "headlight spread pattern" as an individualizing characteristic - NONE. I know that there are cases where this technique has been used, but rhetoric is not science. Many jurisdictions, such as California and Georgia, will allow just about everything in at trial, so not having one's testimony excluded at trial is not proof of anything scientific.

Taking your CCTV footage, you've made your make / model / year determination using the SWGDE's form. Now, how do you move to an individual truck?

This is where basic statistics and inferential reasoning are quite necessary. Do you have sufficient nominal resolution to pick out identifying characteristics in the footage? If not, you're done. The data supports no conclusion as to individualization.

But assuming that you do, how do you work scientifically and as bias free as possible? Unpack the biasing information that you received from your "client" and design an experiment. In the US, given our Constitutional provisions that the accused are innocent until proven guilty, it is for the prosecution to prove guilt. Thus, the staring point for your experiment is that the truck in question is not a match. With sufficient nominal resolution, you set about to prove that there is a match. If you can't, there is no match as far as you're concerned. Remember, the comparative analysis should not be influenced by any other factors or items of evidence.

In designing the experiment, you'll need a sample set of images. You see, a simple "match / no match" comparison needs an adequate sample. It perverts the course of justice to simply attempt the comparison on the accused's vehicle. We don't do witness ID line-ups with just the suspect. Neither should anyone attempt a comparison with just a single "unknown" image - the accused's. Yes, I do use this specific provision of English Common Law to explain the problem here. Perverting the Course of Justice can be any of three acts, fabricating or disposing of evidence, intimidating or threatening a witness or juror, intimidating or threatening a judge. In this case, one Perverts the Course of Justice when one fabricates a conclusion (scientific evidence) where none is possible.

Back to the experiment. How many "unknown" images would you need to approach 99% confidence in your results, thus assisting the course of justice? Answer = 52. How did I come up with 52?

Exact - • Generic binomial test
Analysis: A priori: Compute required sample size
Input: Tail(s)                   = One
Proportion p2 = 0.8
α err prob = 0.01
Power (1-β err prob)     = .99
Proportion p1             = 0.5
Output: Lower critical N = 35.0000000
Upper critical N         = 35.0000000
Total sample size         = 52
Actual power             = 0.9901396
Actual α = 0.008766618

A generic binomial test is similar to the flip of a coin - only two possible outcomes, heads / tails or match / no match. It's the simplest test to perform.


The error probability is your chance of being wrong. At 52 test images, you've got a 1 in 100 chance of being wrong (.99). As you move below 15 test images, you have a greater chance of being wrong than being right. With a sample size of 1, you're likely more accurate tossing a coin.

The 52 samples help us to get to make / model / year. You may chose to refresh those samples with new ones to perform a "blind comparison," and attempt to "include" the suspect's vehicle in your findings. To do this, you'd need the specific description of the "known" vehicle that makes it unique vs the others in the sample.

If I were performing a make / model / year determination, and then a comparison, I would note any errors or limitations in my report. If the data supported no conclusion, or if the limitations in the data prevented me from arriving at a determination, I would note that the data supported no conclusion. If I was able to make a determination, I would have noted my process and how I arrived at the conclusion (in a reliable, valid, and reproducible fashion).

The problem with the reporting of the case is the "cannot be excluded" portion is in the headline. One has to read deeper into the article to find, "... Liscio denied  (that his conclusions may have been formed to fit the bias of the prosecution, who was paying him...), and reminded McGee more than once that he had not identified the truck specifically as Merritt’s..."

Which requires another question be asked, if the analyst had not identified the vehicle, what was he doing there in testimony?

"Among the items that helped to reach the conclusion that the vehicle was “consistent” with Merritt’s truck was a glint caught by the video that matched the position of a latch on a passenger-side storage box toward the rear of the truck, said Liscio,who uses 3D imagery."

Here we move from "cannot be excluded" to "consistent with," another equivocation. How does one not identify a vehicle, but find that said unknown vehicle is "consistent with" the "known" vehicle? This is the problem with Demonstrative Comparisons. When you place a single "known" against a single "unknown" in a demonstrative exhibit, you are making a choice as to what to include in your exhibit - thus you have concluded.

Back to the demonstrative. What is it about the latch on the side of the truck that is unique? Won't all work trucks of this type have latches on their cargo containers? Why is this one so special that it can only be found on the accused's truck? Of these questions, the article does not give an answer.

"I’m not saying that this your client’s vehicle,” Liscio repeated. “All I am saying is that the vehicle in question is consistent with my report, and if there is another vehicle that looks similar, that is possible.” How about at least 326 vehicles found on just two used car web sites?

If you'd like to explore these topics in depth, I'd invite you to sign up for any one (or all) of our upcoming training sessions. Our Statistics for Forensic Analysts course is offered on-line as micro learning and thus enrollment can happen at your convenience. Our other courses can be facilitated at your location or at ours, in Henderson, NV.

Monday, May 20, 2019

What new certification program?

We had a lively discussion in last week's Advanced Processing Techniques class. This class seems to bring out these types of conversations.

We were discussing certifications in our discipline: digital / multimedia analysis. When I said that I know of three, the students seemed puzzled. Three? They were aware of LEVA's CFVT and CFVA programs and the IAI's CFVE program. LEVA counts as one, the IAI, two. Where's three?

Whilst you were not paying attention, a group of analysts got together with the Electronics Technicians Association, International, to create the Audio Video Forensic Analyst (AVFA) certification. Hat's off to them for pulling that one off.

Who is the ETA-i? What is the AVFA? Let's take a look.

The ETA-i is a legit organization that certifies electricians, cable splicers, and other comms techs. It's been around since 1978 and has over 90 certification programs. I spoke with the office manager there about the AVFA certification and received an interesting answer about it's genesis. I was told that a few years ago, they began an expansion into other related industries. They added a Digital Video (DVE) Editor certification. From there, they added the AVFA certification. Some of those involved in the DVE certification were involved in the creation of the AVFA program.

Before you grab your pitch fork and start marching towards the town square, let's take a quick look at the program and compare it to what we know about the existing market for certifications.

Validity is defined as the extent to which an assessment accurately measures what it is intended to measure. Validity refers to the accuracy of measurement. In this case, the assessment is the certification exam. Determining validity can be viewed as constructing an evidence-based argument regarding how well a tool measures what it is supposed to do. Evidence can be found in content, response process, relationships to other variables, and consequences (source).

Reliability refers to whether an assessment instrument gives the same results each time it is used in the same setting with the same type of subjects.  For test/retest reliability, the test should give the same results each time, assuming there are no interval changes in what you are measuring, and they are often measured as correlation.

Face Validity - the extent to which a test is subjectively viewed as covering the concept it purports to measure.

Is the AVFA reliable? As a multiple choice exam, it is certainly possible. One would hope that it's written in such a way that you'd answer the same questions in the same way on successive administrations of the test.

Does the AVFA certification test have face validity? You can find out what it's supposed to measure on it's web site (link). You might look at the competencies sheet and note that there are programs listed that you're likely not familiar with. But, every analysis requires a platform. The creators of the AVFA had to choose tools and terminology. They seem to have chosen freeware and sources of information not hidden behind paywalls. Good for them. The ASTM's glossary is great, but how much do you have to pay to use it?

You can sign up and take the test yourself. No membership required. The test is affordable ($100 + proctor fees). The test is entirely multiple choice. Pass it, you're certified. Fail it, re-take it for free. But, on the face of it (pun intended), it seems to have face validity.

Accredited? Yes. The program at ETA-i is accredited by International Certification Accreditation Council (ICAC). Who are they? I didn't know either, so I called them as well. I had a nice and long conversation with a representative who explained their program and their long relationship with the ETA-i. According to them, they follow ISO 17011 / 17024 in the administration of their accreditation program. So far so good.

What's the deal with accreditation? One of the things that ASTM E2659-09 as well as ISO 17011 / 17024 emphasize is the separation of the training event / training provider and the certifying body. There is a risk of bias if the training provider also runs the certification process - as in teaching the test. This is a large part of why you take Photoshop classes from me, George, Dorothy, Scott Kelby, and etc. ... then trek on over to ProMetric to take the Adobe Certified Expert exam.

This contrasts with LEVA, as an example. LEVA is both training provider and certifying body. As such, it does not follow or comply with ASTM E2659-09 as well as ISO 17011 / 17024 in the design of its program. Thus, it's program is not accredited. There's no way to simply sit for an objective certification exam with LEVA. You must take their classes, all of them, in order to sit for the test. Additionally, it's impossible to judge the reliability or validity of the certification exam as all LEVA certification exams are unique experiences - and are closed to the public. I'm not saying that there's no value to the experience, just that it's not valid from a design / standards standpoint.

The IAI is in the middle. I'm aware that they are seeking accreditation for their programs. They are not a training provider, per se. They do offer training at their conferences, but you can't call that "training to competency." It's more like information sessions, as the ASTM defines training events. Thus, it is possible for the IAI to get their programs accredited. They should, however, remove the discriminatory membership requirement in order to sit for their test.

Thus, as someone who is trained / educated in curriculum and instructional design, who has created many assessments, and has reviewed countless others, I'm excited to see how this new program shakes out. I'll likely sit for the exam soon, just to see what it's like.

If you decide to take the test, let me know what you think of it. Try to go into the process with an open mind. I'm a big fan of democratizing science. If this is a legit certification, I'm all for it.