On November 22, 2005, the Science, State, Justice, Commerce, and Related Agencies Appropriations Act of 2006 became law. Under the terms of the statute, Congress authorized “the National Academy of Sciences to conduct a study on forensic science, as described in the Senate report."
In 2009, the National Academies Press published Strengthening Forensic Science in the United States: A Path Forward (aka, the NAS Report). The report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
On page 208 of the NAS Report, the authors address the issue of certification.
"The certification of individuals complements the accreditation of laboratories for a total quality assurance program. In other realms of science and technology, professionals, including nurses, physicians, professional engineers, and some laboratorians, typically must be certified before they can practice. The same should be true for forensic scientists who practice and testify. Although the accreditation process primarily addresses the management system, technical methods, and quality of the work of a laboratory (which includes the education and training of staff), certification is a process specifically designed to ensure the competency of the individual examiner."
The NAS Report goes on to note, "In addition to improving quality, certification programs can enhance the credibility of certificate holders."
As I noted last week, there are three certifications available in the discipline of digital / multimedia analysis. I would like to correct that here first, leaving the original post unedited.
Of the domains of what is traditionally considered Image Analysis / Video Analysis, there is no specific component within the LEVA / IAI that addresses the testing of a practitioner's competency in authentication. Photogrammetry is sometimes covered at their conventions, but there again, it's not a specified component of the certification programs.
I can say this with certainty about the LEVA certification. One may stand for certification upon successfully completing the LEVA Levels courses, the Courtroom Prep class, and some electives. For the exam, the analyst chooses a case that they've worked and presents it to the certification board. Thus, what is covered in that "exam" is unique to the analyst - varying from analyst to analyst. As a certificate program, it seems quite out of alignment with ASTM E2659 – 09, Standard Practice for Certificate Programs.
As to the domains under examination, there is no list available from either group. The IAI has a reading list that consists of a few older texts as well as some SWGDE guidelines. It's assumed that the practitioner will gain proficiency in these areas via vendor training, attending workshops at conferences, and other training opportunities.
What's missing: audio.
In Tx vs Oliver, the LEVA CFVA's testimony notes the importance of the audio track of the body-worn video recording, and how it helped in the analysts work. But, the CFVA exam does not specifically test for audio competency. I'm certainly not knocking the analyst. Just noting that the CFVA exam doesn't specifically test for competency in audio analysis and the analyst noted how important the audio track was in guiding his work and his conclusions.
Yes, forensic multimedia analysis includes everything in the multimedia container - audio, video, images, metadata, error, and junk. The CFVA - video analyst - seems to just cover the video portion of that list. Not necessarily a bad thing, but it's important to know what it does and doesn't cover.
Let's say you have an "image" from an iPhone - a Live Photo. Live Photos contain sound (link). Does your chosen software platform recognize that a Live Photo is essentially a video file? Does your chosen software platform have the ability to play the audio? What about "enhancing" the audio? Do you know how to enhance audio?
Here's where the AVFA certification has a leg up. Whilst the "competency requirements" aren't all that well written, and the format of the test (as noted on their site) does not match the language in the comps document, the analyst will definitely have an idea as to what will be on the test.
Yes, the AVFA exam includes audio. It's the first A in the acronym. The AVFA exam includes Photogrammetry and Authentication. There doesn't appear to be anything related to comparisons or content analysis, but those domains can sometimes be blended into the others, or generalized to "analysis."
Back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." It seems that the AVFA meets that definition.
I'm going to take the test soon. I'll let you know how that goes as well as comment on the whole process - from sign-up to notification of results and beyond.
Showing posts sorted by relevance for query NAS Report. Sort by date Show all posts
Showing posts sorted by relevance for query NAS Report. Sort by date Show all posts
Wednesday, May 29, 2019
Friday, May 11, 2018
Report writing in forensic multimedia analysis
You've analyzed evidence. You've made a few notes along the way. You've turned those notes over to the process. Your agency doesn't have a specific requirement about what should be in your notes or your report or how detailed they should be. In all the cases that you've worked, you've never been asked for specifics / details.
Now, your case has gone to trial. An attorney is seeking to qualify you to provide expert (opinion) testimony. They introduce you, your qualifications, and what you've been asked to do. The judge may or may not declare you to be an expert so that your opinion can be heard.
As a brief aside, your title or job description can vary widely. I've been an analyst, specialist, director, etc. FRE Rule 702, and the similar rule in your state's evidence code, governs your testimonial experience. Here's the bottom line: according to evidence code, you're not an "expert" unless the Judge says so, and then only for the duration of your testimony in that case. After you're dismissed, you go back to being an analyst, specialist, etc. You may have specific expertise, and that's great. But the assignment of the title of "expert" as relates to this work is generally done by the judge in a specific case, related to the type of testimony that will be offered.
A technician generally offers testimony about a procedure and the results of the procedure. No opinion is given. "I pushed the button and the DVR produced these files."
An expert generally offers opinion based testimony about the results of an experiment or test. "I've conducted a measurement experiment and in my opinion, the unknown subject in the video at the aforementioned date/time is 6’2” tall, with an error of ..."
Everything's OK ... until it's not. You've been qualified as an expert. Is your report ready for trial? What should be in a report anyway?
First off, there's two types of guidance in answering this question. The first type, people's experiences, might help. But, then again, it might not. Just because someone got away with it, doesn't make it a standard practice. Just because you've been through a few trials doesn't make your way "court qualified." These are marketing gimmicks, not standard practices. The second type, a Standard Practice, comes from a standards body like the ASTM. As opposed to the SWG's, who produce guidelines (it would be nice if you ...), standards producing bodies like the ASTM produce standards (you must/shall). For the discipline of Forensic Multimedia Analysis, there are quite a few standards which govern our work. Here's a few of the more important ones:
Did your retrieval follow E1188-11? Did your preparation of the evidence items follow E860-07? Did you assign a unique identifier to each evidence item and label it according to E1459-13? Does your workplace handle evidence according to E1492-11? Did your work on the evidence items follow E2825-12?
If you're not even aware of these standards, how will you answer the questions under direct / cross examination?
Taking a slight step back, and adding more complexity, you're engaged in a forensic science discipline. You're doing science. Science has rules and requirements as well. A scientist's report, in general, is structured in the same way. Go search scientific reports and papers in Google Scholar or ProQuest. The contents and structure of the reports you'll find are governed by the accredited institution. I've spent the last 8 years working in the world of experimental science, conducting experiments, testing data, forming conclusions, and writing reports. The structure for my work was found in the school's guidance documentation and enforced by the school's administrative staff.
How do we know we're doing science? Remember the NAS Report? The result of the NAS Report was the creation of the Organization of Scientific Area Committees for Forensic Science about 5 years ago. The OSAC has been hard at work refining guidelines and producing standards. Our discipline falls within the Video / Image Technology and Analysis (VITAL) Subcommittee. In terms of disclosure, I've been involved with the OSAC since it's founding and currently serve as the Video Task Group Chair within VITAL. But, this isn't an official statement by/for them. Of course, it's me (as me) trying to be helpful, as usual. :)
Last year, an OSAC group issued a new definition of forensic science that can be used for all forensic science disciplines. Here it is:
Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context. Source: A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence. OSAC Task Group on Digital/Multimedia Science. 2017
What is a trace? A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system.
Thus it is that we're engaged in science. Should we not structure our reports in the same way, using the available guidance as to how they should look? Of course. But what would that look like?
Let's assume that your report has a masthead / letterhead with your/your agency's name and contact information. Here's the structure of a report that (properly completed) will conform to the ASTM standards and the world of experimental science.
Administrative Information
Examiner Information
Requestor Information
Unique Evidence Control Number(s)
Chain of Custody Information
Summary of Request
Service Requested (e.g. photogrammetry, authentication, change of format, etc.)
Methodology
Equipment List
Experimental Design / Proposed Workflow
Limitations / Delimitations
Delimitations of the Experiment
Limitations in the Data
Personnel Delimitations / Limitations
Processing
Amped FIVE Processing Report can be inserted here as it conforms to ASTM 2825-12(17).
Results / Summary
Problems / Errors Encountered
Validation
Conclusions
List of Output File(s) / Derivatives / Demonstratives
Approval(s)
Examiner
Reviewer
Administrative Approval
It would generally conclude with a declaration and a signature. Something like this, perhaps:
I, __________, declare under penalty of perjury as provided in 28 U.S.C. §1746 that the foregoing is true and correct, that it is made based upon my own personal knowledge, and that I could testify to these facts if called as a witness.
Now, let's talk about the sections.
The Administrative section.
Methodology
Limitations / Delimitations
Now, your case has gone to trial. An attorney is seeking to qualify you to provide expert (opinion) testimony. They introduce you, your qualifications, and what you've been asked to do. The judge may or may not declare you to be an expert so that your opinion can be heard.
As a brief aside, your title or job description can vary widely. I've been an analyst, specialist, director, etc. FRE Rule 702, and the similar rule in your state's evidence code, governs your testimonial experience. Here's the bottom line: according to evidence code, you're not an "expert" unless the Judge says so, and then only for the duration of your testimony in that case. After you're dismissed, you go back to being an analyst, specialist, etc. You may have specific expertise, and that's great. But the assignment of the title of "expert" as relates to this work is generally done by the judge in a specific case, related to the type of testimony that will be offered.
A technician generally offers testimony about a procedure and the results of the procedure. No opinion is given. "I pushed the button and the DVR produced these files."
An expert generally offers opinion based testimony about the results of an experiment or test. "I've conducted a measurement experiment and in my opinion, the unknown subject in the video at the aforementioned date/time is 6’2” tall, with an error of ..."
First off, there's two types of guidance in answering this question. The first type, people's experiences, might help. But, then again, it might not. Just because someone got away with it, doesn't make it a standard practice. Just because you've been through a few trials doesn't make your way "court qualified." These are marketing gimmicks, not standard practices. The second type, a Standard Practice, comes from a standards body like the ASTM. As opposed to the SWG's, who produce guidelines (it would be nice if you ...), standards producing bodies like the ASTM produce standards (you must/shall). For the discipline of Forensic Multimedia Analysis, there are quite a few standards which govern our work. Here's a few of the more important ones:
- E860-07. Standard Practice for Examining And Preparing Items That Are Or May Become Involved In Criminal or Civil Litigation
- E1188-11. Standard Practice for Collection and Preservation of Information and Physical Items by a Technical Investigator
- E1459-13. Standard Guide for Physical Evidence Labeling and Related Documentation
- E1492-11. Standard Practice for Receiving, Documenting, Storing, and Retrieving Evidence in a Forensic Science Laboratory
- E2825-12(17). Standard Guide for Forensic Digital Image Processing
Did your retrieval follow E1188-11? Did your preparation of the evidence items follow E860-07? Did you assign a unique identifier to each evidence item and label it according to E1459-13? Does your workplace handle evidence according to E1492-11? Did your work on the evidence items follow E2825-12?
If you're not even aware of these standards, how will you answer the questions under direct / cross examination?
Taking a slight step back, and adding more complexity, you're engaged in a forensic science discipline. You're doing science. Science has rules and requirements as well. A scientist's report, in general, is structured in the same way. Go search scientific reports and papers in Google Scholar or ProQuest. The contents and structure of the reports you'll find are governed by the accredited institution. I've spent the last 8 years working in the world of experimental science, conducting experiments, testing data, forming conclusions, and writing reports. The structure for my work was found in the school's guidance documentation and enforced by the school's administrative staff.
How do we know we're doing science? Remember the NAS Report? The result of the NAS Report was the creation of the Organization of Scientific Area Committees for Forensic Science about 5 years ago. The OSAC has been hard at work refining guidelines and producing standards. Our discipline falls within the Video / Image Technology and Analysis (VITAL) Subcommittee. In terms of disclosure, I've been involved with the OSAC since it's founding and currently serve as the Video Task Group Chair within VITAL. But, this isn't an official statement by/for them. Of course, it's me (as me) trying to be helpful, as usual. :)
Last year, an OSAC group issued a new definition of forensic science that can be used for all forensic science disciplines. Here it is:
Forensic science is the systematic and coherent study of traces to address questions of authentication, identification, classification, reconstruction, and evaluation for a legal context. Source: A Framework to Harmonize Forensic Science Practices and Digital/Multimedia Evidence. OSAC Task Group on Digital/Multimedia Science. 2017
What is a trace? A trace is any modification, subsequently observable, resulting from an event. You walk within the view of a CCTV system, you leave a trace of your presence within that system.
Thus it is that we're engaged in science. Should we not structure our reports in the same way, using the available guidance as to how they should look? Of course. But what would that look like?
Let's assume that your report has a masthead / letterhead with your/your agency's name and contact information. Here's the structure of a report that (properly completed) will conform to the ASTM standards and the world of experimental science.
Administrative Information
Examiner Information
Requestor Information
Unique Evidence Control Number(s)
Chain of Custody Information
Summary of Request
Service Requested (e.g. photogrammetry, authentication, change of format, etc.)
Methodology
Equipment List
Experimental Design / Proposed Workflow
Limitations / Delimitations
Delimitations of the Experiment
Limitations in the Data
Personnel Delimitations / Limitations
Processing
Amped FIVE Processing Report can be inserted here as it conforms to ASTM 2825-12(17).
Results / Summary
Problems / Errors Encountered
Validation
Conclusions
List of Output File(s) / Derivatives / Demonstratives
Approval(s)
Examiner
Reviewer
Administrative Approval
It would generally conclude with a declaration and a signature. Something like this, perhaps:
I, __________, declare under penalty of perjury as provided in 28 U.S.C. §1746 that the foregoing is true and correct, that it is made based upon my own personal knowledge, and that I could testify to these facts if called as a witness.
Now, let's talk about the sections.
The Administrative section.
- You're the examiner. If you have help, or someone helped you in your work, they should be listed too. Co-workers, subcontractors, etc.
- The requestor is the case agent, investigator, or the client. The person who asked you to do the work.
- Every item of evidence must have a unique identifier.
- Every item received must be controlled and it's chain of custody tracked. If others accessed the item, their names would be in the evidence control report / list. DEMS and cloud storage solutions like Evidence.com can easily do this and produce a report.
Summary of Request
- What was it that you were asked to do, in plain terms. For example, "Given evidence item #XXX, for date/time/camera, I was asked to determine the vehicle's make/model/year" - comparative analysis / content analysis. Or, "Given evidence item #XXX, for date/time/camera, I was asked to estimate the unknown subject's height" - photogrammetry. Or, "Given image evidence item #XXY-X, retrieved from evidence item #XXY (see attached report), I was asked to determine if the image's contextual information had been altered" - authentication.
- Provide an abstract of the test and the results - a brief overview of what was done and what the results were (with references to appropriate page numbers).
Methodology
- What tools did you use - hardware / software? You may want to include a statement as to each and their purpose / fitness for that purpose. As an example, I use Amped Five. Amped Five is fit for the purpose of conducting image science experiments as it is operationalized from peer-reviewed / published image science. It's processing reports include the source documentation.
- Your proposed workflow. What will guide your work? Can you document it easily? Does your processing report follow this methodology? Hint, it should. Here's my workflow for Photogrammetry, Content Analysis, and Comparative Analysis. You can find it originally in my book, Forensic Photoshop. It's what I use when I work as an analyst. It's what I teach.
Limitations / Delimitations
- Delimitations are the bounds within which your work will be conducted. I will test the image. I won't test the device that created the image.
- With DME, there are a ton of limitations in the data. If the tested question is, what is license plate, and a macro block analysis determines that there is no original data in the area of the license plate, then that is a limitation. If the tested question is, what is the speed of the vehicle, and you don't have access to the DVR, then that is a huge limitation. Limitations must be stated.
- Personnel issues should also be listed. Did someone else start the work that you completed? Was another person employed on the case for a specific reason? Did something limit their involvement? If the question involves the need to measure camera height at a scene, and you can't climb a ladder so you mitigated that in some way, list it.
Remember, on cross examination, attorneys rarely ask questions of people blindly. They likely already know the answer and are walking your down a very specific path to a very specific desired conclusion. Whilst an attorney might not subpoena Verint's tech support staff / communications, as an example, they may have access to the FVA list and may be aware of your communications about the case there. You may not have listed that you received help from that source, but the opposing counsel might. You won't know who's watching what source. They may ask if you've received help on the case. How would you answer if you didn't list the help and disclose the communications, all of the communications? If your agency's policy prohibits the release of case related info, and you shared case related info on the FVA list, your answer to the question now involves specific jeopardy for your career. I've been assigned to Internal Affairs, I've been an employee rep, I know how the system works when one has been accused of misconduct. How do you avoid the jeopardy? Follow your agency's policies and keep good records of your case activity.
Processing
Results / Summary
Approval(s)
Processing
- These are the steps performed and the settings used. This section should read like a recipe so that some other person with similar training / equipment can reproduce your work. This is the essence of Section 4 of ASTM 2825. Amped FIVE Processing Report can be inserted here as it conforms to ASTM 2825-12(17).
Results / Summary
- Did you encounter any problems or errors. List them.
- How did you validate your results? Did anyone peer review your work? This can include test/retest or other such validity exams.
- Conclusions - your opinion goes here. This is the result of your test / experiment / analysis.
- List of Output File(s) / Derivatives / Demonstratives
Approval(s)
- Examiner (your name here), along with anyone else who's work is included in the report.
- Reviewer(s) - was your completed work reviewed? Their name(s).
- Administrative Approval - did a supervisor approve of the completed exam?
Do your reports look like this? Does the opposing counsel analyst's report look like this? If not, why not? It may be an avenue to explore on cross examination. It's best to be prepared.
I know that this is a rather long post. But, I wanted to be rather comprehensive in presenting the topic and list the sources for the information listed. Hopefully, this proves helpful.
Enjoy.
Friday, September 25, 2009
Caution Recommended With Bitemark Evidence
From Medical News Today: Caution Recommended With Bitemark Evidence In Forensics
"Against the backdrop of last week's Congressional hearing into the future of forensic science, researchers from the University at Buffalo's Laboratory for Forensic Odontology Research in the School of Dental Medicine, have published a landmark paper on the controversial topic of bitemark analysis.
The Congressional hearing focused on the findings of a National Academy of Sciences (NAS) report on the scientific basis of forensic disciplines. Among the pattern evidence fields (fingerprints, tool marks, etc.) that were reviewed in the NAS report, bitemark analysis received critical commentary. During the hearing, Innocence Project co-founder Peter Neufeld introduced Roy Brown, wrongfully convicted on bitemark evidence and later exonerated through DNA analysis.
In anticipation of the NAS report, the new UB study published in the Journal of Forensic Sciences challenges the commonly held belief that every bitemark can be perpetrator identified.
"Bitemark identification is not as reliable as DNA identification," explains the study's lead author Raymond G. Miller, D.D.S., UB clinical associate professor of oral diagnostic sciences.
"With DNA, the probability of an individual not matching another can be calculated," he says. "In bitemark analysis, there have been few studies that looked at how many people's teeth could have made the bite."
Read the rest of the story by clicking here.
Enjoy.
"Against the backdrop of last week's Congressional hearing into the future of forensic science, researchers from the University at Buffalo's Laboratory for Forensic Odontology Research in the School of Dental Medicine, have published a landmark paper on the controversial topic of bitemark analysis.
The Congressional hearing focused on the findings of a National Academy of Sciences (NAS) report on the scientific basis of forensic disciplines. Among the pattern evidence fields (fingerprints, tool marks, etc.) that were reviewed in the NAS report, bitemark analysis received critical commentary. During the hearing, Innocence Project co-founder Peter Neufeld introduced Roy Brown, wrongfully convicted on bitemark evidence and later exonerated through DNA analysis.
In anticipation of the NAS report, the new UB study published in the Journal of Forensic Sciences challenges the commonly held belief that every bitemark can be perpetrator identified.
"Bitemark identification is not as reliable as DNA identification," explains the study's lead author Raymond G. Miller, D.D.S., UB clinical associate professor of oral diagnostic sciences.
"With DNA, the probability of an individual not matching another can be calculated," he says. "In bitemark analysis, there have been few studies that looked at how many people's teeth could have made the bite."
Read the rest of the story by clicking here.
Enjoy.
Thursday, September 9, 2010
The challenges continue
From the Belleville News Democrat: "Defense attorneys will attack the bloody fingerprint evidence that prosecutors say connects Darrell "Rell" Lane to the triple stabbing murders of a west Belleville hairdresser and his two clients.
Lane, now 20, appearing in court with his lawyers Ethan Skaggs, Andrew Liefer and LaToya Berry, filed a motion Tuesday asking St. Clair County Circuit Judge Milton Wharton to limit testimony regarding bloodstain pattern analysis and asking for clarification regarding testimony by state's witness David W. Carter, an Illinois State Police blood pattern expert.
Wharton will hear arguments regarding the bloody fingerprint evidence and other pretrial motions on Oct. 11. Lane's trial scheduled to begin Nov. 1.
Lane is accused of three counts of first-degree murder in connection with the stabbing deaths of hair stylist Michael Cooney and his two clients, sisters Doris Fischer, 79, and Dorothy Bone, 82. They were found stabbed to death on March 2, 2005, in Cooney's home-based salon at 7813 W. Main St.
The defense has asked prosecutors Jim Piper and Jon Allard to provide them with Carter's resume', including his education and training, as well as a list of other cases Carter has testified in, plus other details, including video or audio recordings, of any experiments Carter conducted in this case.
In the motion, Liefer cited a National Academy of Science report that found the bloodstain analysis "suffers from the worst deficiencies that impair the reliable determination of scientific truth."
Click here to the read the rest of the story.
The defense bar is taking the NAS report as gospel. In a future series, I'll tackle my response to the NAS report and what we're planning to do if certain scenarios come to pass. The good news is that the Forensic Photoshop workflow remains one of the only peer-reviewed and widely published workflows for the use of Photoshop in court matters.
Enjoy.
Lane, now 20, appearing in court with his lawyers Ethan Skaggs, Andrew Liefer and LaToya Berry, filed a motion Tuesday asking St. Clair County Circuit Judge Milton Wharton to limit testimony regarding bloodstain pattern analysis and asking for clarification regarding testimony by state's witness David W. Carter, an Illinois State Police blood pattern expert.
Wharton will hear arguments regarding the bloody fingerprint evidence and other pretrial motions on Oct. 11. Lane's trial scheduled to begin Nov. 1.
Lane is accused of three counts of first-degree murder in connection with the stabbing deaths of hair stylist Michael Cooney and his two clients, sisters Doris Fischer, 79, and Dorothy Bone, 82. They were found stabbed to death on March 2, 2005, in Cooney's home-based salon at 7813 W. Main St.
The defense has asked prosecutors Jim Piper and Jon Allard to provide them with Carter's resume', including his education and training, as well as a list of other cases Carter has testified in, plus other details, including video or audio recordings, of any experiments Carter conducted in this case.
In the motion, Liefer cited a National Academy of Science report that found the bloodstain analysis "suffers from the worst deficiencies that impair the reliable determination of scientific truth."
Click here to the read the rest of the story.
The defense bar is taking the NAS report as gospel. In a future series, I'll tackle my response to the NAS report and what we're planning to do if certain scenarios come to pass. The good news is that the Forensic Photoshop workflow remains one of the only peer-reviewed and widely published workflows for the use of Photoshop in court matters.
Enjoy.
Sunday, September 6, 2009
Foolproof Forensics?
From NPR.org, commenting on the NAS report:
The world of forensic science has been in turmoil for the last six months since a prestigious panel released a study raising serious questions about many forensic techniques from hair analysis to fingerprints.
Those techniques have collectively resulted in thousands of people landing in prison — and now groups within the forensic science community are fighting over what the next steps should be.
Physicist Thomas Bohan says scientists knew for years that many forensic techniques "lacked scientific evaluation," but there was no political will to do anything about it. Now the report by the National Academy of Sciences offers "an opening, an opportunity," Bohan says.
"It will be a terrible shame if change doesn't take place," says Bohan, who is president of the American Academy of Forensic Sciences.
An Argument Over Next Steps
But every time there is a push for change, there are political battles about what kind of change is best. The debate over forensic science is no different.
Crime lab directors want bigger budgets and more staff. Scientists want to start with research into which forensic techniques are valid.
Some key recommendations from the people who wrote the report don't have much support from anyone. For example, the report recommends creating an independent organization to oversee forensic techniques.
"There is no entity like this right now," says Constantine Gatsonis, who co-chaired the committee that wrote the forensic sciences report. "And hence what you've seen is every entity pulling in [its] own way. My personal opinion is that real progress is going to be very difficult without such an entity."
But that proposal now looks all but dead.
Scott Burns, executive director of the National District Attorneys Association, says his members "don't agree with setting up a national institute of forensic science, another bureaucracy."
Congress will make these decisions. And a Senate Judiciary Committee staffer — speaking on condition of anonymity because the matter is still under discussion — said that in this economic climate it seems unlikely the government will create a new body to oversee forensic science.
The world of forensic science has been in turmoil for the last six months since a prestigious panel released a study raising serious questions about many forensic techniques from hair analysis to fingerprints.
Those techniques have collectively resulted in thousands of people landing in prison — and now groups within the forensic science community are fighting over what the next steps should be.
Physicist Thomas Bohan says scientists knew for years that many forensic techniques "lacked scientific evaluation," but there was no political will to do anything about it. Now the report by the National Academy of Sciences offers "an opening, an opportunity," Bohan says.
"It will be a terrible shame if change doesn't take place," says Bohan, who is president of the American Academy of Forensic Sciences.
An Argument Over Next Steps
But every time there is a push for change, there are political battles about what kind of change is best. The debate over forensic science is no different.
Crime lab directors want bigger budgets and more staff. Scientists want to start with research into which forensic techniques are valid.
Some key recommendations from the people who wrote the report don't have much support from anyone. For example, the report recommends creating an independent organization to oversee forensic techniques.
"There is no entity like this right now," says Constantine Gatsonis, who co-chaired the committee that wrote the forensic sciences report. "And hence what you've seen is every entity pulling in [its] own way. My personal opinion is that real progress is going to be very difficult without such an entity."
But that proposal now looks all but dead.
Scott Burns, executive director of the National District Attorneys Association, says his members "don't agree with setting up a national institute of forensic science, another bureaucracy."
Congress will make these decisions. And a Senate Judiciary Committee staffer — speaking on condition of anonymity because the matter is still under discussion — said that in this economic climate it seems unlikely the government will create a new body to oversee forensic science.
Read the rest of the story by clicking here.
Thursday, April 4, 2019
The "NAS Report," 10 years later
It's been a bit over 10 years since Strengthening Forensic Science in the United States, a Path Forward was released (link). What's changed since then?
As a practitioner and an educator, Chapter 8 was particularly significant. Here's what we should all know - here's what we're all responsible for knowing.
"Forensic Examiners must understand the principles, practices, and contexts of science, including the scientific method. Training should move away from the reliance on the apprentice-like transmittal of practices to education ..."
10 years later, has the situation changed? 10 years later, there are a ton of "apprentice-like" certification programs and just a handful of college programs. College programs are expensive and time consuming. Mid-career professionals don't have the time to sit in college classes for years. Mid-career professionals don't have the money to pay for college. Those that have retired from public service and "re-entered" their profession on the private side face the same challenges.
Years ago, I designed a curriculum set for an undergraduate education in the forensic sciences. It was designed such that it could form the foundation for graduate work in any of the forensic sciences - with electives being being discipline specific. I made the rounds of schools. I made the rounds of organizations like LEVA. I made my pitch. I got nowhere with it. Colleges are non-profit in name only, it seems.
To address the problems of time and money, as well as proximity, I've moved the box of classes on-line. The first offering is out now - Statistics for Forensic Analysts. It's micro learning. It's "consumer level," not "egg head" level. It's paced comfortably. It's priced reasonably. It's entirely relevant to the issues raised in the NAS Report, as well as the issues raised in this month's edition of Significance Magazine.
I encourage you to take a look at Statistics For Forensic Analysts. If you've read the latest issue of Significance, and you have no idea what they're talking about or why what they're talking about is vitally important, you need to take our stats class. Check out the syllabus and sign-up today.
As a practitioner and an educator, Chapter 8 was particularly significant. Here's what we should all know - here's what we're all responsible for knowing.
"Forensic Examiners must understand the principles, practices, and contexts of science, including the scientific method. Training should move away from the reliance on the apprentice-like transmittal of practices to education ..."
10 years later, has the situation changed? 10 years later, there are a ton of "apprentice-like" certification programs and just a handful of college programs. College programs are expensive and time consuming. Mid-career professionals don't have the time to sit in college classes for years. Mid-career professionals don't have the money to pay for college. Those that have retired from public service and "re-entered" their profession on the private side face the same challenges.
Years ago, I designed a curriculum set for an undergraduate education in the forensic sciences. It was designed such that it could form the foundation for graduate work in any of the forensic sciences - with electives being being discipline specific. I made the rounds of schools. I made the rounds of organizations like LEVA. I made my pitch. I got nowhere with it. Colleges are non-profit in name only, it seems.
To address the problems of time and money, as well as proximity, I've moved the box of classes on-line. The first offering is out now - Statistics for Forensic Analysts. It's micro learning. It's "consumer level," not "egg head" level. It's paced comfortably. It's priced reasonably. It's entirely relevant to the issues raised in the NAS Report, as well as the issues raised in this month's edition of Significance Magazine.
I encourage you to take a look at Statistics For Forensic Analysts. If you've read the latest issue of Significance, and you have no idea what they're talking about or why what they're talking about is vitally important, you need to take our stats class. Check out the syllabus and sign-up today.
Wednesday, November 21, 2012
Colleges - the future site of forensic labs?
I've been thinking about this topic ever since the NAS report was issued, can forensic labs function well under police supervision? This article from India hints at our future here in the US - given budget shortfalls, staff cutbacks, and crime rates - can police departments keep up with forensics?
Sure, DNA is the sexy CSI science. But what about latent prints, crime scene photography, or multi-media forensics? As budgets shrink, governments can't afford to back fill open spots in their technical labs. The result, evidence isn't processed.
I've been saying that I think the solution is regional forensic labs at college campuses. Having the "official" lab at a college satisfies a number of issues raised in the NAS report, as well as by Senator Leahy. I think that this model solves a number of problems - more cops out of the labs and back on the street, disinterested third parties practicing forensic science, getting the CSI stuff away from fluctuating city budgets and under a federal funding scheme, and etc.
Just something to think about for now. But, something's got to give ...
Sure, DNA is the sexy CSI science. But what about latent prints, crime scene photography, or multi-media forensics? As budgets shrink, governments can't afford to back fill open spots in their technical labs. The result, evidence isn't processed.
I've been saying that I think the solution is regional forensic labs at college campuses. Having the "official" lab at a college satisfies a number of issues raised in the NAS report, as well as by Senator Leahy. I think that this model solves a number of problems - more cops out of the labs and back on the street, disinterested third parties practicing forensic science, getting the CSI stuff away from fluctuating city budgets and under a federal funding scheme, and etc.
Just something to think about for now. But, something's got to give ...
Monday, December 16, 2019
Independence Matters
Independence matters. When learning new materials, it's important to get an unvarnished view of the information. Too many "official" vendor-sponsored training sessions aren't at all training to competency - they're information sessions, at best, or marketing sessions, at worse.
Vendors generally don't want their employees and contractors to point out the problems with their tools. But as a practitioner, you'll want to know the limitations of that thing you're learning. You'll want to know what it can do, what it does well, and what it doesn't do - or does wrong.
Now, I understand that when people start showing you their resume, they've generally lost the argument. But, in this case, it's an important point of clarification. I do so to show you the value proposition of independent training providers like me. Having been trained and certified by California's POST for curriculum development and instruction means that I understand the law enforcement context. Yes, I was a practitioner who happened to be in police service. The training by POST takes that practitioner information and focuses it, refines it, towards creating and delivering valid learning events. Having an MEdID - a Masters of Education in Instructional Design, means that I've proven that I know how to design instructional programs that will achieve their instructional goals. The PhD in Education is just the icing on the cake, but proves that I know how to conduct research and report my results - an important skillset for evaluating the work of others.
Unlike many of the courses on offer in this space, the courses that I've designed and deliver for the forensic sciences are "competency based." I'm a member of the International Association for Continuing Education and Training (IACET) and follow their internationally recognized Standard for Competency Based Learning.
According to the IACET's research, the Competency-Based Learning (CBL) standard helps organizations to:
This is why, when delivering "product training," I've always wrapped a complete curriculum around the product. I've never just taught FIVE, for example. I've introduced the discipline of forensic digital / multimedia analysis, using the platform (FIVE) to facilitate the work. This is what competency based learning looks like.
Yes, you need to know what buttons to push, and in what order, but you also need to know when it's not appropriate to use the tool and when, how, and why to report inclusive results. You'll need to know how to facilitate a "technical review" and package your results for an independent "peer review." You'll also need to know that there may be a better tool for a specific job - something a manufacturer's representative won't often share.
Align learning with critical organizational imperatives - I've studies the 2009 NAS report. I understand it's implications. The organization that the analyst serves is the Trier of Fact, not the organization that issues their pay. The courses that I've designed and deliver match not only with the US context, but also align with the standards produced in the UK, EU, India, Australia, and New Zealand (many other countries choose to align their national standards with one of these contexts).
Vendors generally don't want their employees and contractors to point out the problems with their tools. But as a practitioner, you'll want to know the limitations of that thing you're learning. You'll want to know what it can do, what it does well, and what it doesn't do - or does wrong.
Now, I understand that when people start showing you their resume, they've generally lost the argument. But, in this case, it's an important point of clarification. I do so to show you the value proposition of independent training providers like me. Having been trained and certified by California's POST for curriculum development and instruction means that I understand the law enforcement context. Yes, I was a practitioner who happened to be in police service. The training by POST takes that practitioner information and focuses it, refines it, towards creating and delivering valid learning events. Having an MEdID - a Masters of Education in Instructional Design, means that I've proven that I know how to design instructional programs that will achieve their instructional goals. The PhD in Education is just the icing on the cake, but proves that I know how to conduct research and report my results - an important skillset for evaluating the work of others.
Unlike many of the courses on offer in this space, the courses that I've designed and deliver for the forensic sciences are "competency based." I'm a member of the International Association for Continuing Education and Training (IACET) and follow their internationally recognized Standard for Competency Based Learning.
According to the IACET's research, the Competency-Based Learning (CBL) standard helps organizations to:
- Align learning with critical organizational imperatives.
- Allocate limited training dollars judiciously.
- Ensure learning sticks on the job.
- Provide learners with the tools needed to be agile and grow.
- Improve organizational outcomes, return on mission, and the bottom line.
This is why, when delivering "product training," I've always wrapped a complete curriculum around the product. I've never just taught FIVE, for example. I've introduced the discipline of forensic digital / multimedia analysis, using the platform (FIVE) to facilitate the work. This is what competency based learning looks like.
Align learning with critical organizational imperatives - I've studies the 2009 NAS report. I understand it's implications. The organization that the analyst serves is the Trier of Fact, not the organization that issues their pay. The courses that I've designed and deliver match not only with the US context, but also align with the standards produced in the UK, EU, India, Australia, and New Zealand (many other countries choose to align their national standards with one of these contexts).
Allocate limited training dollars judiciously - Whilst most training vendors continue to raise prices, I've dropped mine. I've even found ways to deliver the content on-line as micro learning, to get prices as low as possible. Plus, it helps your agency meet it's sustainability goals.
Ensure learning sticks on the job - Building courses for all learning styles helps to ensure that you'll remember, and apply, your new knowledge and skills.
Provide learners with the tools needed to be agile and grow - this is another reason for building on-line micro learning offerings. Learning can happen when and where you're available, and where and when you need it.
Improve organizational outcomes, return on mission, and the bottom line - competency based training events help you improve your processing speed, which translates to more cases worked per day / week / month. This is your agency's ROM and ROI. Getting the work done right the first time is your ROM and ROI.
I'd love to see you in a class soon. See for yourself what independence means for learning events. You can come to us in Henderson, NV. We've got seats available for our upcoming Introduction to forensic multimedia analysis with Amped FIVE sessions (link). You can join our on-line offering of that course (link), saving a ton of money doing so. Or, we can come to you.
Have a great day, my friends.
keywords: audio forensics, video forensics, image forensics, audio analysis, video analysis, image analysis, forensic video, forensic audio, forensic image, digital forensics, forensic science, amped five, axon five, training, forensic audio analysis, forensic video analysis, forensic image analysis, amped software, amped software training, amped five training, axon five training, amped authenticate, amped authenticate training
keywords: audio forensics, video forensics, image forensics, audio analysis, video analysis, image analysis, forensic video, forensic audio, forensic image, digital forensics, forensic science, amped five, axon five, training, forensic audio analysis, forensic video analysis, forensic image analysis, amped software, amped software training, amped five training, axon five training, amped authenticate, amped authenticate training
Friday, May 31, 2019
Is it practical?
Continuing the discussion of certification in general, the NAS Report notes the following about certification programs, "The American Bar Association has recommended that certification standards be required of examiners, including “demanding written examinations, proficiency testing, continuing education, recertification procedures, an ethical code, and effective disciplinary procedures.” Let's break these down.
Demanding written exams. What does this mean? Are "written exams" composed of essay questions where one explains topics in depth? Are "written exams" objective tests, like multiple choice exams? What makes an exam "demanding?"
Could an examination body survey the entirety of my written work, books, papers, speeches, this blog, and so on? Could they judge my proficiency based on my written work? I'm not sure. But, pulling from my academic life, it is entirely possible for someone to be proficient in their craft, but be non-verbal. How do you conduct a "demanding written exam" on a non-verbal person? What about differences of language? Do we insist upon all tests being in US English? How do we handle the issue of testing vulnerable populations? Given my extensive work in the autistic community, both academically and professionally, these are relevant questions.
Proficiency testing. Does this belong to an independent certification program or within a vendor's training program? For example, who is a better judge of my proficiency with my chosen tools, Amped Software, Inc. (vendor) or the IAI (third party)? What does the IAI know about FIVE and Authenticate? Nothing, actually. They're still focussed on Photoshop. How do I know this? Their reading list has George Reis' Photoshop book as one of the sources for the test. Whilst still relevant, and quite good, it's a 10 year old treatise on George's favorite PS techniques in no particular order. Given that I no longer use Photoshop in my analytical workflow, how is a test on Photoshop helpful in assessing my proficiency as an examiner?
Perhaps the "requirement" could be met with a review of an analyst's case documentation. Submit a few reports, have the panel examine them in light of the ASTM and SWGDE standards / guidelines.
Continuing education. There comes a point in time, when you've been doing this for a while, that you run out of classes to take. Some, that are available, are cost prohibitive. That's a large part of why I've moved a lot of the classes that I teach to an LMS as micro-learning. As an example, if you can't get in to your local college to take a statistics course, you can take Statistics for Forensic Analysts on-line at an affordable price. Same for redaction, Redaction for Standards Compliance is offered on-line featuring a variety of software platforms.
I've been around the US and Canada quite a bit. I realize that traveling to training is not always an option. I think the internet offers some amazing options to level the playing field as regards continuing education. The question becomes, will your chosen certification program accept your completion certificate from an on-line course or provider?
Recertification procedures. I think the important part here is not only that the individual circle back and re-certify, but that the certification program continually monitors the industry and keeps the certification program up to date. Four years from now, when my AVFA certification is due for renewal, I should not be presented with the same test instrument. Laws and technology change. So too should the certification program. Again, I love George Reis to death, but is his book still relevant in a modern multimedia workflow? Perhaps. But are there other, more appropriate texts? Yes. Absolutely. What about listing Dr. John Russ' Image Processing Cookbook as well, given the preference for Photoshop? Or his Image Processing Handbook, for a more academic look at image processing? If price is an issue, they could choose Fundamentals of Digital Image Processing by Dr. Anil Jain. It's a text book that can be bought used for under $10.
Ethical code. Here's LEVA's Code of Ethics. This post is an example of article 5 in LEVA's COE, offering constructive criticism. Interestingly, article 9 is what prevented me from ever being LEVA certified. Article 9 states, "An Analyst shall maintain the confidentiality of evidence received and work undertaken in accordance with any such request by the submitting agency." It was the policy of the Los Angeles Police Dept. that case work remain confidential and no evidence leave the secure facility without permission. When I sought permission to begin the process of certification, it was flatly denied. Given the LEVA certification methodology, if you can't present one of your cases, you can't get certified. Thus, in honoring their COE, I was prevented from ever being certified whilst in police service. Now, in retirement and in private practice, LEVA discriminates against me for not being a current public service employee by charging confiscatory prices for their offerings.
For the IAI's COE, click here. It begins on pg 79 of the linked document.
Effective disciplinary procedures. One of the things that we do is review opposing counsel's analysis. The examiner on the other side of any case should expect a thorough review of not only their work but their credentials as submitted through the discovery process. I have lost track of the amount of times I have encountered embellishments and fabrication of credentials by analysts, some of which have been certified by LEVA or the IAI. Everything you write in your report should be factual. Everything you write about yourself should be verifiable. As an example of this, I note as a matter of fact that I am retired from police service, having served just shy of 15 years with the Los Angeles Police Dept. under civil service classification 3687. This is a verifiable statement. I can provide the job description for civil service classification 3687, or you can click on the link in the previous sentence. I can provide my retiree ID. I could even print out an account statement from the City's retirement plan noting how many years/months I have of retirement credits (an indication of years/months worked). But, if I simply quit my job, it is a slightly different process than retiring. Thus, I wouldn't say that I had retired, but that I had quit / separated / moved on - or some other term that would accurately describe the end of my time in police service. Same goes for my education. No embellishment, just a simple presentation of the facts.
Back to ethics, there have been a few times that I reported my findings to a certification body. To the best of my knowledge, nothing was ever done with the information that I provided. This is the problem with most professional organizations. We're all friends. No one wants to judge their friends. No waves. No trouble. Let it slide. If there are no consequences, there's no effective discipline.
As these all relate to the certification experience that I just undertook, the AVFA includes all elements except the "proficiency test." I was not given an item of evidence and asked to examine it vs. a question under enquiry. Could my body of work stand in a proficiency test's stead? How much have I written? How many courses of instruction have I designed? How many have I trained to competency? How many cases have I worked? After all, it's this body of work that prepared me to take and pass the exam.
Getting back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." My education, training, and experience prepared me to take and pass the examination that was prepared by peers, thus demonstrating to the certificate program a minimum level of competence. As to the peers who prepared the exam, I may not know a single one of them, but having a list of names with contact information is a great start.
Demanding written exams. What does this mean? Are "written exams" composed of essay questions where one explains topics in depth? Are "written exams" objective tests, like multiple choice exams? What makes an exam "demanding?"
Could an examination body survey the entirety of my written work, books, papers, speeches, this blog, and so on? Could they judge my proficiency based on my written work? I'm not sure. But, pulling from my academic life, it is entirely possible for someone to be proficient in their craft, but be non-verbal. How do you conduct a "demanding written exam" on a non-verbal person? What about differences of language? Do we insist upon all tests being in US English? How do we handle the issue of testing vulnerable populations? Given my extensive work in the autistic community, both academically and professionally, these are relevant questions.
Proficiency testing. Does this belong to an independent certification program or within a vendor's training program? For example, who is a better judge of my proficiency with my chosen tools, Amped Software, Inc. (vendor) or the IAI (third party)? What does the IAI know about FIVE and Authenticate? Nothing, actually. They're still focussed on Photoshop. How do I know this? Their reading list has George Reis' Photoshop book as one of the sources for the test. Whilst still relevant, and quite good, it's a 10 year old treatise on George's favorite PS techniques in no particular order. Given that I no longer use Photoshop in my analytical workflow, how is a test on Photoshop helpful in assessing my proficiency as an examiner?
Perhaps the "requirement" could be met with a review of an analyst's case documentation. Submit a few reports, have the panel examine them in light of the ASTM and SWGDE standards / guidelines.
Continuing education. There comes a point in time, when you've been doing this for a while, that you run out of classes to take. Some, that are available, are cost prohibitive. That's a large part of why I've moved a lot of the classes that I teach to an LMS as micro-learning. As an example, if you can't get in to your local college to take a statistics course, you can take Statistics for Forensic Analysts on-line at an affordable price. Same for redaction, Redaction for Standards Compliance is offered on-line featuring a variety of software platforms.
I've been around the US and Canada quite a bit. I realize that traveling to training is not always an option. I think the internet offers some amazing options to level the playing field as regards continuing education. The question becomes, will your chosen certification program accept your completion certificate from an on-line course or provider?
Recertification procedures. I think the important part here is not only that the individual circle back and re-certify, but that the certification program continually monitors the industry and keeps the certification program up to date. Four years from now, when my AVFA certification is due for renewal, I should not be presented with the same test instrument. Laws and technology change. So too should the certification program. Again, I love George Reis to death, but is his book still relevant in a modern multimedia workflow? Perhaps. But are there other, more appropriate texts? Yes. Absolutely. What about listing Dr. John Russ' Image Processing Cookbook as well, given the preference for Photoshop? Or his Image Processing Handbook, for a more academic look at image processing? If price is an issue, they could choose Fundamentals of Digital Image Processing by Dr. Anil Jain. It's a text book that can be bought used for under $10.
Ethical code. Here's LEVA's Code of Ethics. This post is an example of article 5 in LEVA's COE, offering constructive criticism. Interestingly, article 9 is what prevented me from ever being LEVA certified. Article 9 states, "An Analyst shall maintain the confidentiality of evidence received and work undertaken in accordance with any such request by the submitting agency." It was the policy of the Los Angeles Police Dept. that case work remain confidential and no evidence leave the secure facility without permission. When I sought permission to begin the process of certification, it was flatly denied. Given the LEVA certification methodology, if you can't present one of your cases, you can't get certified. Thus, in honoring their COE, I was prevented from ever being certified whilst in police service. Now, in retirement and in private practice, LEVA discriminates against me for not being a current public service employee by charging confiscatory prices for their offerings.
For the IAI's COE, click here. It begins on pg 79 of the linked document.
Effective disciplinary procedures. One of the things that we do is review opposing counsel's analysis. The examiner on the other side of any case should expect a thorough review of not only their work but their credentials as submitted through the discovery process. I have lost track of the amount of times I have encountered embellishments and fabrication of credentials by analysts, some of which have been certified by LEVA or the IAI. Everything you write in your report should be factual. Everything you write about yourself should be verifiable. As an example of this, I note as a matter of fact that I am retired from police service, having served just shy of 15 years with the Los Angeles Police Dept. under civil service classification 3687. This is a verifiable statement. I can provide the job description for civil service classification 3687, or you can click on the link in the previous sentence. I can provide my retiree ID. I could even print out an account statement from the City's retirement plan noting how many years/months I have of retirement credits (an indication of years/months worked). But, if I simply quit my job, it is a slightly different process than retiring. Thus, I wouldn't say that I had retired, but that I had quit / separated / moved on - or some other term that would accurately describe the end of my time in police service. Same goes for my education. No embellishment, just a simple presentation of the facts.
Back to ethics, there have been a few times that I reported my findings to a certification body. To the best of my knowledge, nothing was ever done with the information that I provided. This is the problem with most professional organizations. We're all friends. No one wants to judge their friends. No waves. No trouble. Let it slide. If there are no consequences, there's no effective discipline.
As these all relate to the certification experience that I just undertook, the AVFA includes all elements except the "proficiency test." I was not given an item of evidence and asked to examine it vs. a question under enquiry. Could my body of work stand in a proficiency test's stead? How much have I written? How many courses of instruction have I designed? How many have I trained to competency? How many cases have I worked? After all, it's this body of work that prepared me to take and pass the exam.
Getting back to the NAS Report, "In essence, ‘certification’ usually means that a particular individual has completed a defined course of education, training, and experience, and has passed an examination prepared by peers which demonstrates that the individual has obtained at least the minimum level of competence required to practice the specific discipline." My education, training, and experience prepared me to take and pass the examination that was prepared by peers, thus demonstrating to the certificate program a minimum level of competence. As to the peers who prepared the exam, I may not know a single one of them, but having a list of names with contact information is a great start.
Friday, March 5, 2010
AAFS weighs in on NAS report
From the Seattle Times:
"Crime science tackles tainted image at Seattle meeting
The overriding issue at the American Academy of Forensic Sciences meeting in Seattle this week isn't likely to find its way into a "CSI" television script. In the wake of a blistering report on the nation's crime labs, forensic experts are trying to shore up the scientific credentials of many of their workhorse techniques.
A National Academy of Sciences (NAS) panel concluded last year that analysis of bite marks, blood spatters, handwriting and even fingerprints is not backed by the type of rigorous evidence that is standard in other scientific disciplines.
"The dominant message here ... is that the emperor really doesn't have all his clothes on," said Donald Kennedy, former president of Stanford University and an organizer of the NAS review.
Bohan said most forensic scientists have taken that message to heart.
The White House Office of Science and Technology Policy established a forensic-science subcommittee, and legislation will be introduced in Congress next month to bolster research and oversight of crime labs. But Bohan is impatient for progress.
"Everybody is talking about what to do," he said.
The Department of Justice (DOJ) is beginning to fund fundamental research in several areas, including ballistics and fire-debris analysis, said Michael Sheppo, leader of the forensic sciences at DOJ's research arm."
Click here for the rest of the story.
Enjoy.
"Crime science tackles tainted image at Seattle meeting
The overriding issue at the American Academy of Forensic Sciences meeting in Seattle this week isn't likely to find its way into a "CSI" television script. In the wake of a blistering report on the nation's crime labs, forensic experts are trying to shore up the scientific credentials of many of their workhorse techniques.
A National Academy of Sciences (NAS) panel concluded last year that analysis of bite marks, blood spatters, handwriting and even fingerprints is not backed by the type of rigorous evidence that is standard in other scientific disciplines.
"The dominant message here ... is that the emperor really doesn't have all his clothes on," said Donald Kennedy, former president of Stanford University and an organizer of the NAS review.
Bohan said most forensic scientists have taken that message to heart.
The White House Office of Science and Technology Policy established a forensic-science subcommittee, and legislation will be introduced in Congress next month to bolster research and oversight of crime labs. But Bohan is impatient for progress.
"Everybody is talking about what to do," he said.
The Department of Justice (DOJ) is beginning to fund fundamental research in several areas, including ballistics and fire-debris analysis, said Michael Sheppo, leader of the forensic sciences at DOJ's research arm."
Click here for the rest of the story.
Enjoy.
Thursday, May 30, 2019
Taking the AVFA Certification Exam
I have to say that signing up and taking the AVFA certification exam was a rather pleasant experience. Yes, that seems a bit odd to say. But, it's true. It should be true that an organization treats you like an adult, respectfully walking you through the process of engaging their services.
I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.
You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.
I can't emphasize this enough, I paid $120 in fees to take this certification exam.
If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.
For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.
At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.
Back to the experience.
I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.
The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.
The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.
In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certificate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.
Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.
I started on the ETA-i web site. This is where one pays the exam fees, the first stop in the process. Simply select the "Pay Exam Fees" option in their store, select the exam, and pay the fee.
You can do a bit of scouting as to a testing location ahead of your purchase, and include that information in the purchasing process. It will speed things along a bit. I selected a Prometric testing close by. After I paid the exam fees, ETA-i sent my information and the exam to the testing center. I then paid the proctoring fees to the testing center. In total, I paid $120 in fees.
I can't emphasize this enough, I paid $120 in fees to take this certification exam.
If you're sitting for the LEVA CFVA board, you will have to get there (transportation costs) and stay there at least a few days (hotel and food). Likely, you'll want to stay the whole week and, assuming you pass, get your certificate at the Friday dinner. Assuming average travel costs, you or your agency will spend about $2000 for this experience. This $2000 comes after spending thousands upon thousands in the Levels courses and the electives.
For the IAI's CFVE, the cost of testing is significantly lower than LEVA. The IAI charges $400 for the initial test. Many complain about the process of registering for the exam - the letters of recommendation, the documentation of training, and so forth. Plus, finding a proctor location can also be an issue. You can look at their list of certified examiners here. There are only 22 people on the list. Considering the previous post, and the NAS report's admonition about the need for certified analysts, 22 examiners does not even come close to addressing the need. Then, you can see that there are only 9 new examiners since the original group passed through the gates. That's not sustainable.
At $100, the ETA-i's AVFA is affordable. Many agencies do not support their employees in certification. Mine didn't. Everything I did for LEVA, NaTIA, the IRIT, FISWG, and the OSAC, I did on my own time (vacation / comp time). My decision to not get LEVA certification was as much financial as it was practical - I couldn't afford to go out of pocket on certification given the fact that my employer didn't care if I was certified or not. It wasn't a condition of employment and California does not require it for testimony.
Back to the experience.
I paid my fees to the ETA-i and to the testing center. I made an appointment to take the test and showed up a bit early to fill out the paperwork and sign all the forms. The testing center was just that, a quiet place full of cubicles and people taking a variety of tests. The proctor verified my identity. But the proctor, and the ETA-i, did not ask me about my employer or if I work primarily for "the defense." It shouldn't matter for whom I work or what types of cases I typically process. For the AVFA, the price is the price. No membership fees. No tiered pricing for government / non-government. No defense vs prosecution bias (in truth, I probably spent more time defending my self-insured public agency employer than working as a part of prosecuting criminal cases). That bias has always bugged me.
The test instrument was delivered on time and I had 2 hours to answer 100 questions. A comfortable amount of time for me, given my almost 2 decades of experience in the discipline. Upon completion, I notified the proctor who collected all of the materials and wished me well. I returned to my car and to my day. I wasn't stressed out at all. I was prepared to take the assessment and I was treated respectfully by all parties involved. In all, it was probably one of the more pleasant testing experiences I've experienced.
The testing center does not score the test. That's done back at the ETA-i offices. Thus, there's a bit of a delay in finding out if you've passed. For me, I had the results back in less than a day. I passed the test. In the next week or so, I'll receive the official letter notifying me of the results as well as the actual certificate. They also include a wallet card - perhaps necessary if confronted about one's status.
In thinking back over this whole experience, the ETA-i is separate from all of the training that I've received over the years, as is the testing center. Thus, it seems to comply with the ASTM's E2659 – 09. Standard Practice for Certificate Programs. Their accreditation provider follows ISO/IEC 17024 in providing oversight of the ETA-i's programs. This for me is a big deal. I want to know that the certificate that I just earned is legitimate. Given my experiences with NaTIA and LEVA on retirement, I also want to know that in the transition from police service to private practice I'll be treated the same. I was treated with respect throughout the entire process with the ETA-i.
Given the reach of the ETA-i, as well as the availability of proctoring sites from companies like Prometric, the AVFA seems poised to help the NAS Report's recommendations come to fruition - if there's a certification in your discipline, you should have it. The AVFA's certification is well within the reach (financially) of the average analyst / examiner. It's certainly worth a look.
Friday, October 28, 2011
Leahy Bill? Don't hold your breath
I normally don't wade into the turbulent world of politics, but I am going to today ... sort of. I attended a presentation recently, and (reading between the lines) I think that those who believe that the Leahy Bill (Criminal Justice and Forensic Science Reform Act) is dead in the water and nothing is going to happen are going to be in for a big surprise.
Meet The Executive Office of the President of the United States' Subcommittee on Forensic Science (SoFS).
According to their web site, "The purpose of the Subcommittee on Forensic Science (SoFS) is to advise and assist the Committee on Science (COS), the National Science and Technology Council (NSTC), and other coordination bodies of the Executive Office of the President on policies, procedures and plans related to forensic science in the national security, criminal justice, and medical examiner/coroner systems at the local, state and federal levels."
Ok. So what do they want to do with their office space? Good question. Here's what they say are their goals and objectives, "This Subcommittee was created to assess the practical challenges of implementing recommendations in the 2009 National Research Council (NRC) report entitled Strengthening Forensic Science in the United States: A Path Forward, and to advise the White House on how best to achieve the goals outlined in that report. The SoFS is charged with developing practical and timely approaches to enhancing the validity and reliability of the federal government’s forensic science activities. This includes ensuring that regional, state and local entities adopt best practices in forensic sciences, and facilitating a strong coordinated effort across federal agencies to identify and address important federal policy, program and budget matters."
With me so far. No opinion yet ... just straight reporting from their web site.
Begin opinion: Look at the purpose statement. Their purpose is to advise on forensic science as it relates to criminal justice (among other things) at the federal, state, and local levels. Their goals include advising the White House on how best to implement the recommendations contained in the NAS report. Now read this statement again, "This includes ensuring that regional, state and local entities adopt best practices in forensic sciences, and facilitating a strong coordinated effort across federal agencies to identify and address important federal policy, program and budget matters." How does the federal government assure compliance in other areas of policy? With money. Again, my opinion and my observations are thus - soon enough you'll be faced with a choice; implement the recommendations or face the loss of federal funds. No need for Congress to act. No need for the Leahy Bill. This will come by way of regulation, enforced with a simple tool (money). Don't believe me? How was "education reform" accomplished? By attaching funding strings to education bills like NCLB. Remember, in a general sense, what the federal government funds, it controls.
Please don't get me wrong. Best Practices are necessary. Consensus standards are a good thing. Everyone in our business should be working from an SOP. Things like quality assurance/control, peer-review, training, and so on are great. How many police labs accredit their DME forensic functions? Not many. How many would be ready to accredit them soon? Not many. How many agencies could withstand the loss of some/all federal funds if their DME forensic functions were not under their main accreditation umbrella? Not many.
In doing my part to help things go right, I'm working with LEVA to create an accreditation program for DME labs. It will serve as an alternative for those agencies who are not ASCLD/LAB accredited and are performing DME forensics. More on this later.
Again, while my commentary may be a bit political in tone, it's more of a "here's the way I see it based on my recent experiences." I'd be interested to hear how you see this working, or how you see this affecting your agency.
Enjoy.
Meet The Executive Office of the President of the United States' Subcommittee on Forensic Science (SoFS).
According to their web site, "The purpose of the Subcommittee on Forensic Science (SoFS) is to advise and assist the Committee on Science (COS), the National Science and Technology Council (NSTC), and other coordination bodies of the Executive Office of the President on policies, procedures and plans related to forensic science in the national security, criminal justice, and medical examiner/coroner systems at the local, state and federal levels."
Ok. So what do they want to do with their office space? Good question. Here's what they say are their goals and objectives, "This Subcommittee was created to assess the practical challenges of implementing recommendations in the 2009 National Research Council (NRC) report entitled Strengthening Forensic Science in the United States: A Path Forward, and to advise the White House on how best to achieve the goals outlined in that report. The SoFS is charged with developing practical and timely approaches to enhancing the validity and reliability of the federal government’s forensic science activities. This includes ensuring that regional, state and local entities adopt best practices in forensic sciences, and facilitating a strong coordinated effort across federal agencies to identify and address important federal policy, program and budget matters."
With me so far. No opinion yet ... just straight reporting from their web site.
Begin opinion: Look at the purpose statement. Their purpose is to advise on forensic science as it relates to criminal justice (among other things) at the federal, state, and local levels. Their goals include advising the White House on how best to implement the recommendations contained in the NAS report. Now read this statement again, "This includes ensuring that regional, state and local entities adopt best practices in forensic sciences, and facilitating a strong coordinated effort across federal agencies to identify and address important federal policy, program and budget matters." How does the federal government assure compliance in other areas of policy? With money. Again, my opinion and my observations are thus - soon enough you'll be faced with a choice; implement the recommendations or face the loss of federal funds. No need for Congress to act. No need for the Leahy Bill. This will come by way of regulation, enforced with a simple tool (money). Don't believe me? How was "education reform" accomplished? By attaching funding strings to education bills like NCLB. Remember, in a general sense, what the federal government funds, it controls.
Please don't get me wrong. Best Practices are necessary. Consensus standards are a good thing. Everyone in our business should be working from an SOP. Things like quality assurance/control, peer-review, training, and so on are great. How many police labs accredit their DME forensic functions? Not many. How many would be ready to accredit them soon? Not many. How many agencies could withstand the loss of some/all federal funds if their DME forensic functions were not under their main accreditation umbrella? Not many.
In doing my part to help things go right, I'm working with LEVA to create an accreditation program for DME labs. It will serve as an alternative for those agencies who are not ASCLD/LAB accredited and are performing DME forensics. More on this later.
Again, while my commentary may be a bit political in tone, it's more of a "here's the way I see it based on my recent experiences." I'd be interested to hear how you see this working, or how you see this affecting your agency.
Enjoy.
Thursday, September 15, 2011
Expediency?
--- Begin editorial ---
I was recently contacted by someone who needed me to work on a case involving video based evidence. I examined the evidence briefly and told them that it would take about a week to set up the experiments, test/validate the results, and return a report. His answer, "can it be done quicker? I don't care about the other stuff. I just need to get something that I can use and get it out to the media."
My answer was simple. I turned down the job. I tried to explain to the customer that I'd either do it the right way, or not at all - that justice demands the process be followed and that expediency has no place in science.
Expediency? Expediency is a regard for what is politic or advantageous rather than for what is right or just. "I'm just trying to get this done ..." "Quick and dirty." "It's not for court, it's for the media ..."
If the multimedia evidence will be used to help identify a suspect, either through a BOLO flyer, a media alert, or bill boards and bus benches ... is it really "not for court?"
What if "quick and dirty" gets it wrong? What if, in "just trying to get something done," your freeware tools or shoddy processes ignore aspect, colour and light, drop frames, etc. In the process, you get the shape of the face wrong, the skin or clothing colour wrong ... and someone identifies the wrong person. What if your "quick and dirty" wastes time and resources as investigators look in the wrong direction?
What's our responsibility to the Trier of Fact to get it right, from start to finish? Is it possible to be both fast and accurate? I would argue that it is ... provided that the technician has the appropriate training and equipment.
Is it ethical to base your workflow or business processes on expediency? I've received a number of e-mails asking about speeding up processes because of mandatory filing deadlines. Essentially what's being said is that the analyst isn't being given the proper amount of time because a suspect is in custody and the investigator needs to get the case filed. Ouch.
What about expediency in terms of assignment of personnel? Is it ethical to place an employee into a position for which he/she does not have appropriate training and experience? These are all questions facing supervisors as budgets shrink. What's the appropriate response? Is the NAS report right in that lab functions should be privatized, separated from governmental agencies?
With the Crawford doctrine taking hold and defendants exercising their right to question everyone, attorneys are digging into the training and expertise of technicians who are involved in the prosecution's case. Given Brady, would a technician necessarily want to harm his/her career by being on the losing end of a thorough cross examination - being found unqualified to provide testimony? What role do managers play in preventing this from happening? In other words, if you know that your people are not as qualified as they should be, and you know attorneys are using Crawford (6th Ammendment) to call everyone related to the case, why would you entertain expediency? In "just getting things done," you may be ending the careers of your people and ultimately harming your case.
This is just an editorial on my part. Now that budgets are shrinking, I've been getting so many e-mails on the subject of "getting things done" that I had to say something. I'm not speaking about any one agency or firm - just collectively about the practice of expediency. Expediency has no place in the justice system, in my humble opinion.
I was recently contacted by someone who needed me to work on a case involving video based evidence. I examined the evidence briefly and told them that it would take about a week to set up the experiments, test/validate the results, and return a report. His answer, "can it be done quicker? I don't care about the other stuff. I just need to get something that I can use and get it out to the media."
My answer was simple. I turned down the job. I tried to explain to the customer that I'd either do it the right way, or not at all - that justice demands the process be followed and that expediency has no place in science.
Expediency? Expediency is a regard for what is politic or advantageous rather than for what is right or just. "I'm just trying to get this done ..." "Quick and dirty." "It's not for court, it's for the media ..."
If the multimedia evidence will be used to help identify a suspect, either through a BOLO flyer, a media alert, or bill boards and bus benches ... is it really "not for court?"
What if "quick and dirty" gets it wrong? What if, in "just trying to get something done," your freeware tools or shoddy processes ignore aspect, colour and light, drop frames, etc. In the process, you get the shape of the face wrong, the skin or clothing colour wrong ... and someone identifies the wrong person. What if your "quick and dirty" wastes time and resources as investigators look in the wrong direction?
What's our responsibility to the Trier of Fact to get it right, from start to finish? Is it possible to be both fast and accurate? I would argue that it is ... provided that the technician has the appropriate training and equipment.
Is it ethical to base your workflow or business processes on expediency? I've received a number of e-mails asking about speeding up processes because of mandatory filing deadlines. Essentially what's being said is that the analyst isn't being given the proper amount of time because a suspect is in custody and the investigator needs to get the case filed. Ouch.
What about expediency in terms of assignment of personnel? Is it ethical to place an employee into a position for which he/she does not have appropriate training and experience? These are all questions facing supervisors as budgets shrink. What's the appropriate response? Is the NAS report right in that lab functions should be privatized, separated from governmental agencies?
With the Crawford doctrine taking hold and defendants exercising their right to question everyone, attorneys are digging into the training and expertise of technicians who are involved in the prosecution's case. Given Brady, would a technician necessarily want to harm his/her career by being on the losing end of a thorough cross examination - being found unqualified to provide testimony? What role do managers play in preventing this from happening? In other words, if you know that your people are not as qualified as they should be, and you know attorneys are using Crawford (6th Ammendment) to call everyone related to the case, why would you entertain expediency? In "just getting things done," you may be ending the careers of your people and ultimately harming your case.
This is just an editorial on my part. Now that budgets are shrinking, I've been getting so many e-mails on the subject of "getting things done" that I had to say something. I'm not speaking about any one agency or firm - just collectively about the practice of expediency. Expediency has no place in the justice system, in my humble opinion.
Wednesday, August 7, 2019
Where'd you get the 10?
When the "Search for Images on the Web" functionality was introduced into Amped's Authenticate some time ago, I asked a simple question of the development team, "where'd you get the 10?"
Amped SRL prides itself on operationalizing peer-reviewed published papers in image science. I assumed, wrongly it seems, that there was some science behind the UI's default setting for how many pictures you'd like to find. There isn't.
What I found out, at the time, is that the "Stop After Finding At Least N Pictures. N is" default of 10 is set to 10 for no particular reason whatsoever. My own opinion is that it's set to 10 because the developers are engineers and 10 - a one and a zero - looks nice. The 10 has no foundation in science / statistics as a valid number for that field. If you accept the default as presented, you're creating a convenience sample set that will give you more chances of being wrong than being right. Here's why.
The basic question being tested in the developer's example, with help from the "Search Images From Same Camera Model" dialog, is "match / no match." Does the evidence item match a representative sample of images from the same make / model of camera? You would perform this check when your evidence item's JPEG QT is not found in your tool's internal database (this is a known limitation of all software that is dependent upon an internal database). Another way of framing "match / no match" is a Generic Binomial Test.
Here's what the sample size calculation looks like for a generic binomial test performed in the criminal justice context (as opposed to the university / research context).
A two-tailed test has a better ability to limit Type I and Type II errors vs. a one-tailed test. α and β error probability are set as low as possible, one chance per 100. This protocol yields a recommended sample size of 59. Not 10.
Around 20 samples, you have more chances of being wrong than being right. At 10 samples, you have about 8 chances in 10 of being wrong.
The "match / no match" scenario is quite different than an attempt to establish "ground truth," or what the camera "should be producing" when it creates an image. For these tests, a "performance model" is required and is generated by samples created by the device in question. Each camera will present it's own issues and the results of your test of the camera in question can't be applied to other cameras of the same make / model. You'll need to know a bit about the signal path - from light coming in to the resulting image's storage - to be able to determine the correct value for the "predictors" variable. In my last experiment, that value was 17 ... 17 different parts / processes that could possibly be in error. In that case, the sample size calculation was as shown below.
In this case, I needed to generate 164 valid files for my sample set - not 10. The plot is shown below.
With only 10 samples for such a test, you would have 9 chances in 10 of being wrong.
Another scenario where the "official guidance" on sample sizes is quite off is with PRNU. The official guidance notes that the number of reference images in building a reference pattern is limited to 50, "(the number of images is limited to 50, which proved to be enough – we’ll also discuss this in a future post)." But, from the source documentation (link), the authors recommend more than that, "Obviously, the larger the number of images Np, the more we suppress random noise components and the impact of the scene. "Based on our experiments, we recommend using Np > 50" (link). The authors of the source document actually used 320 samples per camera in proving out their theories - not 10 or 50. Other researchers examining PRNU have used even larger sample sets (link) (link). These three links weren't chosen at random to attempt to reinforce my point. These are the references found in the processing report generated by Amped's Authenticate - further illustrating the point about validation of tools and methodologies. (Many thanks to Dr. Fridrich for maintaining a massive list of source documentation.)
But, as my old college football coach used to say, "the wrong way will work some of the time, the right way will work all of the time." Simply choosing a convenient number of samples may not be a problem in a justice system that has a 95% plea rate. But, as recent news points out, if you get caught out for bad methodology, all of your past work gets re-opened. Don't let this happen to you. Use sound methods. Get educated on the foundations of your work.
Remember, tools like Amped SRL's Authenticate don't render an opinion as to a file's authenticity - you do. Your opinion must be backed by a sound methodology, compliance with standards, and the fundamentals of science.
Correcting the lack of understanding of this vital topic was on the 2009 NAS Report's list of recommendations (link).
"The issues covered during the committee’s hearings and deliberations included:
Amped SRL prides itself on operationalizing peer-reviewed published papers in image science. I assumed, wrongly it seems, that there was some science behind the UI's default setting for how many pictures you'd like to find. There isn't.
What I found out, at the time, is that the "Stop After Finding At Least N Pictures. N is" default of 10 is set to 10 for no particular reason whatsoever. My own opinion is that it's set to 10 because the developers are engineers and 10 - a one and a zero - looks nice. The 10 has no foundation in science / statistics as a valid number for that field. If you accept the default as presented, you're creating a convenience sample set that will give you more chances of being wrong than being right. Here's why.
The basic question being tested in the developer's example, with help from the "Search Images From Same Camera Model" dialog, is "match / no match." Does the evidence item match a representative sample of images from the same make / model of camera? You would perform this check when your evidence item's JPEG QT is not found in your tool's internal database (this is a known limitation of all software that is dependent upon an internal database). Another way of framing "match / no match" is a Generic Binomial Test.
Here's what the sample size calculation looks like for a generic binomial test performed in the criminal justice context (as opposed to the university / research context).
Analysis: A priori: Compute required sample size
Input: Tail(s) = Two
Proportion p2 = 0.8
α err prob = 0.01
Power (1-β err prob) = 0.99
Proportion p1 = 0.5
Output: Lower critical N = 19.0000000
Upper critical N = 40.0000000
Total sample size = 59
Actual power = 0.9912792
Actual α = 0.0086415
A two-tailed test has a better ability to limit Type I and Type II errors vs. a one-tailed test. α and β error probability are set as low as possible, one chance per 100. This protocol yields a recommended sample size of 59. Not 10.
Around 20 samples, you have more chances of being wrong than being right. At 10 samples, you have about 8 chances in 10 of being wrong.
The "match / no match" scenario is quite different than an attempt to establish "ground truth," or what the camera "should be producing" when it creates an image. For these tests, a "performance model" is required and is generated by samples created by the device in question. Each camera will present it's own issues and the results of your test of the camera in question can't be applied to other cameras of the same make / model. You'll need to know a bit about the signal path - from light coming in to the resulting image's storage - to be able to determine the correct value for the "predictors" variable. In my last experiment, that value was 17 ... 17 different parts / processes that could possibly be in error. In that case, the sample size calculation was as shown below.
t tests - Linear multiple regression: Fixed model, single regression coefficient
Analysis: A priori: Compute required sample size
Input: Tail(s) = Two
Effect size f² = 0.15
α err prob = 0.01
Power (1-β err prob) = 0.99
Number of predictors = 17
Output: Noncentrality parameter δ = 4.9598387
Critical t = 2.6099227
Df = 146
Total sample size = 164
Actual power = 0.9900250
In this case, I needed to generate 164 valid files for my sample set - not 10. The plot is shown below.
With only 10 samples for such a test, you would have 9 chances in 10 of being wrong.
Another scenario where the "official guidance" on sample sizes is quite off is with PRNU. The official guidance notes that the number of reference images in building a reference pattern is limited to 50, "(the number of images is limited to 50, which proved to be enough – we’ll also discuss this in a future post)." But, from the source documentation (link), the authors recommend more than that, "Obviously, the larger the number of images Np, the more we suppress random noise components and the impact of the scene. "Based on our experiments, we recommend using Np > 50" (link). The authors of the source document actually used 320 samples per camera in proving out their theories - not 10 or 50. Other researchers examining PRNU have used even larger sample sets (link) (link). These three links weren't chosen at random to attempt to reinforce my point. These are the references found in the processing report generated by Amped's Authenticate - further illustrating the point about validation of tools and methodologies. (Many thanks to Dr. Fridrich for maintaining a massive list of source documentation.)
But, as my old college football coach used to say, "the wrong way will work some of the time, the right way will work all of the time." Simply choosing a convenient number of samples may not be a problem in a justice system that has a 95% plea rate. But, as recent news points out, if you get caught out for bad methodology, all of your past work gets re-opened. Don't let this happen to you. Use sound methods. Get educated on the foundations of your work.
Remember, tools like Amped SRL's Authenticate don't render an opinion as to a file's authenticity - you do. Your opinion must be backed by a sound methodology, compliance with standards, and the fundamentals of science.
Correcting the lack of understanding of this vital topic was on the 2009 NAS Report's list of recommendations (link).
"The issues covered during the committee’s hearings and deliberations included:
- (a) the fundamentals of the scientific method as applied to forensic practice—hypothesis generation and testing, falsifiability and replication, and peer review of scientific publications;
- (b) the assessment of forensic methods and technologies—the collection and analysis of forensic data; accuracy and error rates of forensic analyses; sources of potential bias and human error in interpretation by forensic experts; and proficiency testing of forensic experts;
- (c) infrastructure and needs for basic research and technology assessment in forensic science
- (d) current training and education in forensic science; ..." pg. 3
10 years later, and vendors are still ignoring the NAS' recommendations, often providing incorrect information to customers.
The scientific method forms the foundation of all of the forensic science training offerings that I've created over the years. Illustrating error rates, where they appear in the work, and how to calculate and control for them can be found in all of our forensic science courses.
If you'd like to move beyond "push-button forensics," I hope to see you in class soon.
Tuesday, June 19, 2012
Love is not love which alters when it alteration finds
Let me not to the marriage of true minds
Admit impediments ...
Last year, I introduced you - gentle reader - the President's Subcommittee on Forensic Science. One of the themes of the work from this body, and the NAS report, is that if there's a certification in your field of work ... you should have it.
LEVA has their certifications. Some argue that the value of LEVA's certification is outweighed by the high cost to attain the title, Certified Forensic Video Analyst (CFVA). The IAI now as a competing offering - Certified Forensic Video Examiner. The IAI's path to certification is inexpensive in comparison to LEVA.
I received an e-mail asking about the difference between the two - and the semantical difference between "analyst" and "examiner."
Bing's dictionary defines analyst as an "expert who examines something: somebody with specialist knowledge or skill who studies or examines something by separating it into its constituent parts and gives an assessment, description, or explanation of it." The root of examiner is examine, "to inspect or scrutinize carefully." Given the Subcommittee or NAS, should a person engaged in FVA have either of these, or both? Is one more valuable vs. the other? Should an analyst be thought to outrank or be superior to an examiner? What's in a name?
Before you answer that ... what about Adobe's ACE program? Again, the premise is that if there's a certification, you should have it. Do you use Photoshop or PremierePro? There are separate ACE exams for these products. What about Acrobat? If you make PDF presentations, should you need to be an Acrobat ACE? How about Avid? Avid has a Media Composer certification. If you bought a MC system from Ocean Systems, do you need to be Avid certified?
Herein lies the problem. The goals of the certification programs of Adobe and Avid are not centered on law enforcement. In order to be certified on their products, you will have to master tools and techniques that you may never use in the course of your work. Folks further argue that LEVA's training path is too Avid centric and that the IAI test focusses too much on broadcast TV. You name the test or certification, and there'll be folks with arguments against it. With this in mind, what's the real value to certification?
Yesterday, I said that while I still love Photoshop, I'm falling out of love with it. The manufacturers of Photoshop and the other commercial products that I use are targeting a different audience. If my workflow only uses specific pieces of Photoshop, should there be allowances for this practice in the certified user/expert test? What if the manufacturer of your tool of choice offers no certification?
Just some things to consider. More to follow ...
Admit impediments ...
Last year, I introduced you - gentle reader - the President's Subcommittee on Forensic Science. One of the themes of the work from this body, and the NAS report, is that if there's a certification in your field of work ... you should have it.
LEVA has their certifications. Some argue that the value of LEVA's certification is outweighed by the high cost to attain the title, Certified Forensic Video Analyst (CFVA). The IAI now as a competing offering - Certified Forensic Video Examiner. The IAI's path to certification is inexpensive in comparison to LEVA.
I received an e-mail asking about the difference between the two - and the semantical difference between "analyst" and "examiner."
Bing's dictionary defines analyst as an "expert who examines something: somebody with specialist knowledge or skill who studies or examines something by separating it into its constituent parts and gives an assessment, description, or explanation of it." The root of examiner is examine, "to inspect or scrutinize carefully." Given the Subcommittee or NAS, should a person engaged in FVA have either of these, or both? Is one more valuable vs. the other? Should an analyst be thought to outrank or be superior to an examiner? What's in a name?
Before you answer that ... what about Adobe's ACE program? Again, the premise is that if there's a certification, you should have it. Do you use Photoshop or PremierePro? There are separate ACE exams for these products. What about Acrobat? If you make PDF presentations, should you need to be an Acrobat ACE? How about Avid? Avid has a Media Composer certification. If you bought a MC system from Ocean Systems, do you need to be Avid certified?
Herein lies the problem. The goals of the certification programs of Adobe and Avid are not centered on law enforcement. In order to be certified on their products, you will have to master tools and techniques that you may never use in the course of your work. Folks further argue that LEVA's training path is too Avid centric and that the IAI test focusses too much on broadcast TV. You name the test or certification, and there'll be folks with arguments against it. With this in mind, what's the real value to certification?
Yesterday, I said that while I still love Photoshop, I'm falling out of love with it. The manufacturers of Photoshop and the other commercial products that I use are targeting a different audience. If my workflow only uses specific pieces of Photoshop, should there be allowances for this practice in the certified user/expert test? What if the manufacturer of your tool of choice offers no certification?
Just some things to consider. More to follow ...
Thursday, November 3, 2011
MultiMedia Forensics Innocence Project?
Months ago, Larry E. Daniel posted a call for a Digital Forensics Innocence Project. Referencing another article's comment, ""What we need is a Computers Forensics version of the Innocence Project. We need experts who believe in the presumption of innocence and are willing to spend the time it takes to dig through logs, registry entries and hard drives to find exculpatory material when present. Prosecutors who look for – and presume – guilt do selective searches for data supporting guilt; those accused rarely have the resources to counter such selective evidence," Larry goes on a slightly more moderate tone to say, "I agree with him in principal, considering that there are people who are charged with crimes who do not have the resources to hire experts. And in cases where the client cannot meet the standard to be declared indigent and receive funding for an expert, I believe that we as experts in the field should be willing to take on a reasonable number of pro bono cases."
One of the things presented in the NAS report is the strangle hold that police crime labs have on the forensic science process. Finding an expert that you can afford, who is not part of a police crime lab, can be difficult. The problem is ... we have to get there. There are a few labs out there. Price is driven by a combination of supply/demand, the cost to operate, and the price set by the courts (when they're paying the bill).
I'd love to see some version of the Innocence Project that includes digital multimedia forensics. If it ever comes ... sign me up.
Enjoy.
One of the things presented in the NAS report is the strangle hold that police crime labs have on the forensic science process. Finding an expert that you can afford, who is not part of a police crime lab, can be difficult. The problem is ... we have to get there. There are a few labs out there. Price is driven by a combination of supply/demand, the cost to operate, and the price set by the courts (when they're paying the bill).
I'd love to see some version of the Innocence Project that includes digital multimedia forensics. If it ever comes ... sign me up.
Enjoy.
Wednesday, July 23, 2014
Have politics and personalities influenced decisions in police controlled labs?
I recently received a head's up about an open letter written by an LE commander for whom I used to work.
This letter has stirred up some strong emotions and heated comments. You can read the letter, so without speaking to the specifics of his allegations, it does raise some very interesting questions for LE agency laboratories. If politics and personalities can (allegedly) make their way into internal investigations and sway outcomes, can they also make their way into criminal proceedings?
Remember that the NAS Report called for moving forensic science labs outside of police or prosecutor control. More stories like this will only serve to move us faster toward that eventuality.
This letter has stirred up some strong emotions and heated comments. You can read the letter, so without speaking to the specifics of his allegations, it does raise some very interesting questions for LE agency laboratories. If politics and personalities can (allegedly) make their way into internal investigations and sway outcomes, can they also make their way into criminal proceedings?
Remember that the NAS Report called for moving forensic science labs outside of police or prosecutor control. More stories like this will only serve to move us faster toward that eventuality.
Monday, January 11, 2010
The Digital Forensic Sub-Disciplines?
In the December 2009/January 2010 issue of Forensic Magazine, John J. Barbara posits the following question to begin his article on digital forensic sub-disciplines: "Can we clearly differentiate whether an examination falls under Computer Forensics, Forensic Audio, Image Analysis, or Video Analysis?"
To begin with, I was a little suspicious of the article - given the author's profession. It seemed to me that he is making a pitch here that will certainly help his business. What's his business, you ask? He runs a company that specialises in helping agencies set up and comply with the ASCLD/LAB and ISO regulations. Is there a conflict here? Perhaps - but read the article for yourself before forming any conclusions.
One of the key points of the article (for me) was hidden about midway down the page. "Although the initiative toward gaining accreditation for the discipline was being driven primarily by the federal agencies ..." Why are federal agencies driving the initiative? Are state and local agencies participating in a meaningful way? What about privateers? How are their interests represented?
The author walks the reader through a hypothetical case where a single analyst processes a piece of digital multimedia evidence. Then he proceeds to illustrate that the analyst really was performing four functions (sub-disciplines) as part of the process. The sub-text hints that there is something inherently wrong with the single analyst model.
This article, for me, represents the latest in a worrisome trend in advocacy - away from the local / private lab toward a regional / nationally controlled lab. This, while good for the author of the article's business, it is certainly not good for the citizens of this country. Here's why ...
This trend, the NAS report, the SWG's and so forth, has been moving forensic work in a certain direction. That direction is towards greater standardisation and control of the work. So far so good. But the next logical step is to say that agencies with a single analyst (like many local police agencies and privateers) should no longer be to allowed to work across sub-disciplines. I, for example, have received training and worked cases involving video, image, audio, cell phone, and small device forensics. I also image hard drives (computer forensics). Must I now choose a specialty? What does this do to the local agency, given today's economic mess? The only agencies that are currently hiring analysts in great numbers are federal. But are federal agencies going to handle the video when a car is broken into on my block? Hardly.
Information and standardisation are good when they are used as a guide toward mastery of a given subject. They are bad when they are used to drive the market towards a specific vendor's practice - even when that "vendor" is the government. I'd hate to see the state of justice in this country when hundreds or thousands of well meaning and quite capable privateers and local LE analysts are driven out of business in the face of these "mandates." The right to an vigorous defense will be in jeopardy when all forensic functions are dispensed at the federal or regional level - driven by cost controls and governed by shrinking budgets.
If "forensics" is discussion and debate ... let the best rise practices to the top - be them private, local, state, regional, or national - in an openly competitive atmosphere. When the federal government steps in and mandates these practices - thus driving out perfectly good practitioners - everyone loses. At this point, forensics, as such, is dead. There is no discussion or debate allowed. Everyone loses.
This is just one citizen's opinion. What say you?
To begin with, I was a little suspicious of the article - given the author's profession. It seemed to me that he is making a pitch here that will certainly help his business. What's his business, you ask? He runs a company that specialises in helping agencies set up and comply with the ASCLD/LAB and ISO regulations. Is there a conflict here? Perhaps - but read the article for yourself before forming any conclusions.
One of the key points of the article (for me) was hidden about midway down the page. "Although the initiative toward gaining accreditation for the discipline was being driven primarily by the federal agencies ..." Why are federal agencies driving the initiative? Are state and local agencies participating in a meaningful way? What about privateers? How are their interests represented?
The author walks the reader through a hypothetical case where a single analyst processes a piece of digital multimedia evidence. Then he proceeds to illustrate that the analyst really was performing four functions (sub-disciplines) as part of the process. The sub-text hints that there is something inherently wrong with the single analyst model.
This article, for me, represents the latest in a worrisome trend in advocacy - away from the local / private lab toward a regional / nationally controlled lab. This, while good for the author of the article's business, it is certainly not good for the citizens of this country. Here's why ...
This trend, the NAS report, the SWG's and so forth, has been moving forensic work in a certain direction. That direction is towards greater standardisation and control of the work. So far so good. But the next logical step is to say that agencies with a single analyst (like many local police agencies and privateers) should no longer be to allowed to work across sub-disciplines. I, for example, have received training and worked cases involving video, image, audio, cell phone, and small device forensics. I also image hard drives (computer forensics). Must I now choose a specialty? What does this do to the local agency, given today's economic mess? The only agencies that are currently hiring analysts in great numbers are federal. But are federal agencies going to handle the video when a car is broken into on my block? Hardly.
Information and standardisation are good when they are used as a guide toward mastery of a given subject. They are bad when they are used to drive the market towards a specific vendor's practice - even when that "vendor" is the government. I'd hate to see the state of justice in this country when hundreds or thousands of well meaning and quite capable privateers and local LE analysts are driven out of business in the face of these "mandates." The right to an vigorous defense will be in jeopardy when all forensic functions are dispensed at the federal or regional level - driven by cost controls and governed by shrinking budgets.
If "forensics" is discussion and debate ... let the best rise practices to the top - be them private, local, state, regional, or national - in an openly competitive atmosphere. When the federal government steps in and mandates these practices - thus driving out perfectly good practitioners - everyone loses. At this point, forensics, as such, is dead. There is no discussion or debate allowed. Everyone loses.
This is just one citizen's opinion. What say you?
Subscribe to:
Posts (Atom)