Peer Review?

What is peer-review? What is the role of the peer-reviewer? In reviewing academic / scientific research, is peer-review different than reviewing an academic / scientific work-product?

Let's examine the questions. But first, some definitions.

What is peer-review? As a noun, peer-review is an evaluation of scientific, academic, or professional work by others working in the same field. As a verb, peer-review is the action of subjecting (someone or something) to a peer review.

According to the John Jay College of Criminal Justice [source]: "In academic publishing, the goal of peer review is to assess the quality of articles submitted for publication in a scholarly journal. Before an article is deemed appropriate to be published in a peer-reviewed journal, it must undergo the following process:

• The author of the article must submit it to the journal editor who forwards the article to experts in the field. Because the reviewers specialize in the same scholarly area as the author, they are considered the author’s peers (hence “peer review”).
• These impartial reviewers are charged with carefully evaluating the quality of the submitted manuscript.
• The peer reviewers check the manuscript for accuracy and assess the validity of the research methodology and procedures.
• If appropriate, they suggest revisions. If they find the article lacking in scholarly validity and rigor, they reject it.

Because a peer-reviewed journal should not publish articles that fail to meet the standards established for a given discipline, peer-reviewed articles that are accepted for publication should exemplify the best research practices in a field." Here's a nice explanation of the difference between Scholarly (peer-reviewed) vs Popular articles (link to video).

Thus, peer-review is an evaluation of the accuracy and validity of a paper / article submitted for publication in a scholarly journal performed by an impartial group of anonymous others working in the same field.

How does one perform a review?

How does one check for accuracy and validity? Start with structure. As our field is a science, forensic science, is the paper divided into sections with headings such as those listed below?

• Introduction
• Literature review
• Theory or Background
• Subjects
• Methods
• Results
• Discussion
• Conclusion

Does the article have footnotes or citations of other sources? This questions asks, is it easy for the reader / reviewer to move between the topic and the reference  Does the article have a bibliography or list of references at the end? This question speaks to the organization of references. Each journal will have it's standards in terms of how references are handled. Does the item under review follow the journal / publication's standard (APA, ACS, Harvard, etc.)? If you are reviewing, are you familiar with the formatting rules used by the publication? Are the author's credentials listed? Is the author a subject matter expert? Do their credentials check out?

In reviewing for accuracy, are the referenced sources available to you (the reviewer). If yes, great. Do the references support / reject the author's work? If not, likely they won't be available to the eventual reader. You may wish to suggest to the author that they find source for reference that is more generally available to the audience. An example of this can be seen in authors that cite an obscure source, unavailable to the reader, in an appeal to authority (logical fallacy) to support a novel idea - knowing that the reviewer / reader will have no way of verifying (checking the accuracy) of the assertion.

Reviewing for accuracy is much more than simply checking the author's math. Many of the problems with papers come from the author's mistakes / misuse of referenced materials.

How does a journal or periodical choose peer-reviewers?

The answer to this question is both simple and complex. The short, simple answers is ... no one knows. Leading to the complex answer, it depends. It depends upon the topic, the availability of qualified subject matter experts, the publication schedule vs. the reviewers' schedules, etc. The answers to these questions help you evaluate the quality of the publication.

In evaluating the quality of the publication, are you (the reader) able to access the peer-review criteria? Does the journal publish a list of it's peer-reviewers by topic? In other words, how do you (the reader / consumer of the information) know if a paper has been evaluated by qualified subjected matter experts?

Are there peer-reviewed journals or publications that cover the forensic science domain of forensic multimedia analysis?

Yes. But they are few and far between.

The IAI publishes the Journal of Forensic Identification (JFI) at $205/yr. The JFI deals primarily with issues around latent prints, having only 6 articles that touch multimedia evidence since 2010 (mostly case reports). It's available to members of the IAI and subscribers. If you're an academic and / or have access to the EBSCO Criminal Justice database service or the National Criminal Justice Reference Service, you can find it there. In all, it's not a popular place to find cutting-edge articles in our discipline.

The American Academy of Forensic Sciences publishes the Journal of Forensic Sciences (JFS) at $354/year. Just as the IAI, there are few articles of value to the forensic multimedia analyst. Also like the IAI, if you're not a subscriber or an academic, you'll not be able to access the few editions and articles that are relevant to our discipline.

There's also the Digital Forensic Research Workshop (DFRWS). It doesn't publish a journal, it has workshops around the world. Some of it's papers can be found on it's web site.

In all, there are a lot of problems with the above listed journals in terms of the validity of the articles presented as "research." Two important criteria for judging the adequacy of research are internal validity, the study’s success at establishing causality, and external validity, the study’s generalizability to other settings and other times. For the case studies, no attempt at internal validity is generally offered. The authors simply list a series of events taken in their work. Because of this, the external validity is non-existent. The results of their work can only be applied to their work.

In examining the work found in the above referenced sources, a question emerges: is there a difference between academic / scientific journals and the journals available to forensic multimedia analysts?

The answer, unfortunately, is yes. The work found in the JFI can largely be considered case reports. Case reports do not advance science, they report on an analyst's work on a single case. The analyst might have figured out how to do something, but those results are confined to that specific set of circumstances. Nowhere in the JFI, for example, does another analyst take someone's case notes and attempt an experiment in an attempt to validate the method, or create an experiment that builds out the case information such that the results can be applied to a greater population of cases.

Where do we go from here?

This is a topic that I've been exploring in some depth lately. I work with a research organization that seems like it may want to explore the creation of a journal that publishes novel work, at a greater frequency, and with results available to the greater community without fee. Stay tuned on that ...