Experimental Design

Before one begins any sort of research, one usually surveys the literature on the topic to see if any research has been completed and what, if anything, was concluded. Sure, the researcher has a general idea about what they want to study, but a literature review helps to inform and refine the eventual design of the study. According to Shields and Rangarajan (2013), there's a difference between the process of reviewing the literature and a finished work or product known as a literature review. The process of reviewing the literature is often ongoing and informs many aspects of the empirical research project. See what I just did, I discovered some research on literature reviews, and inserted the summary into my paragraph. Usually, there's an accompanying citation. Here it is: Shields, P., Rangarjan, N. (2013). A Playbook for Research Methods: Integrating Conceptual Frameworks and Project Management. Stillwater, Oklahoma: New Forums Press. ISBN 1-58107-247-3.

I received some feedback about the few posts I've written regarding "headlight spread pattern analysis." One was very intriguing - "... assume the premise is true, that there is uniqueness that can be discovered through experimentation. Where would you begin? What would the experimental design look like?" Hmmm....

Given the term, "headlight spread pattern analysis," there are four distinct elements - "headlamps," "the diffusion of light," "pattern matching," and a methodology for "analysis." Each of these would need to handled separately before adding the next element - the recording of this diffusion of light within a scene.

Let's just do a bit of research on the types of headlamps available to the general commercial market, leaving the other three elements for later.

Our first discovery is that we must separate the "lamp" from the "bulb." The bulb provides the "light" and the "lamp" is a system for the projection of that light.

For the lamp's housing, there are two general types: "reflector" and "projector." Of the light sources ("bulb"), there are several available types: Tungsten, Tungsten-halogen, High-intensity discharge (HID), LED, Laser. Of the "filament" type lamps, there are over 35 types available in for sale in the US, and covered by the World Forum for Harmonization of Vehicle Regulations (ECE Regulations), which develops and maintains international-consensus regulations on light sources acceptable for use in lamps on vehicles and trailers type-approved for use in countries that recognise the UN Regulations.

Given the eventual experimental design, it's important to note that the US and Canada "self-certify" compliance with the ECE Regulations. No prior verification is required by a governmental agency or authorised testing entity before the equipment can be imported, sold, installed, or used.

For a bulb's operation, there are variables to consider. There's voltage (usually 12V) and wattage (between 20w - 75w) - collectively known as "Nominal Power." Then there's "luminous flux." In photometry, luminous flux or luminous power is the measure of the perceived power of light. It differs from radiant flux, the measure of the total power of electromagnetic radiation (including infrared, ultraviolet, and visible light), in that luminous flux is adjusted to reflect the varying sensitivity of the human eye to different wavelengths of light.

Lots of big words there. But two stand out - luminous flux (the measure of the perceived power of light) and radiant flux (the measure of the total power of electromagnetic radiation). For our experiment, we'll need to differentiate between these as someone / some people are going to compare patterns (perception is subjective). We'll also need an objective measure of the total power of our samples. Luminous flux is used in the standard as the point of headlamps is to improve the drivers perception of the scene in front of them as they drive.

Luminous flux is measured in lumens. On our list of bulbs, the luminous flux values are reported as being between 800lm and 1750lm with a tolerance of between +/-10% and +/15%. This makes the range between 680lm and 2012.5lm. It's important to remember that the performance of a bulb over it's life span is not binary (e.g. 1550lm constantly until is stops working). Performance of lamps degrade over time.

Back to the lamp as a system. There are the general types of fixed lamps - they're bolted on to the front of the vehicle by at least four fasteners. These types need to be "aimed" at the time of installation, which can shift over time as the fasteners loosen. There are also "automatic" lamps, which feature some form of "beam aim control." These "beam aim control" types include, headlamp leveling systems, directional headlamps, advanced front-lighting system (AFS), automatic beam switching, Intelligent Light System, adaptive highbeam, and glare-free high beam and pixel light.

Now in the cases that I've reviewed, it seems that "headlight spread pattern analysis" was employed when a proper vehicle make / model determination failed due to a lack of available detail - usually due to a low nominal resolution.

Given what I've just shared above, about the potential variables in our study, an important revelation emerges. If there is insufficient nominal resolution to conduct a vehicle make / model determination, which considers class characteristics like presence / quantity of features like doors and windows before considering the presence / quantity / type of features within those items, then how could there be a determination as to type of lamp system and bulb that would be necessary for any comparison of headlight spread pattern? What if there's a general match of the shape of the pattern, but the quality of light is wrong? Or, what if the recorder's recording process and compression scheme corrupt the shape of the light dispersal or change the quality of the light? How then is a "scientific" comparison possible?

Short answer - it's not. This is one of the ways in which "forensics" (rhetoric) is used to mask a lack of science in "forensic science."

But, let's take a look at that question in a different way. Given all of the variables listed above, what would a normal distribution of "headlight spread patterns" "look like" (observed without recoding) for each of the possible combinations of system, bulb, and mounted position? What would they look like after being recorded on a DVR? This adds more variables to the equation.

For the recording system, there's the camera / lens combination, there's the transmission method, and there's the recorder's frame rate and compression scheme to contend with. Sure, you have the evidence item. But you don't know if the system was operating "normally" during that recording, or what "normal" even is until you produce a performance model of the system's operation in the recording of everything listed above. You'll need the "ground truth" of the recorder's capabilities in order to perform a proper experiment.

Remember, the recording may be "known" - meaning you retrieved it from the system and controlled it's custody such that the integrity of the file is not in question. But, what is unknown is the make / model of the vehicle. THIS CAN'T BE PRESUPPOSED. IT MUST BE DETERMINED.

In the cases that I reviewed, each comparison was performed against a presupposed make / model of vehicle - the "suspect's vehicle." If convenience samples were employed for a "comparison," then it was a few handy cars of the same make / model / year as the accused's vehicle. THIS IS NOT A VEHICLE DETERMINATION. This is no different than a single-person line-up, or a "show-up." This method has no relationship with science.

Back to the literature review and how it may inform a future experimental design.

What I've discovered is that the quantity of variables is quite large. Actually, the quantity of system types, then the variables within those systems, is quite large. This is before considering how these will be recorded by a given recorder. This information would be required to validate case work, aka a CASE STUDY. A case study is only applicable to that one case. If one wanted to validate the technique, then an appropriate amount of recorders would need to be included (proper samples of complete system types).

Given all of this, the cost of a single case study would be beyond the budget of most investigative agencies. It's certainly beyond my budget. The cost of testing the questions, "headlight spread pattern analysis has no validity" (H null) and "headlight spread pattern analysis has validity" (H1) would be massive.

Nevertheless, given all of the above, to conclude "match" - it is "the accused's vehicle" - one must rule out all other potential vehicles. Given that estimates put the number of cars and trucks in the United States at between 250-260 million vehicles for a country with 318 million people, then "match" says "to the exclusion of between 250-260 million vehicles" - which doesn't include the random Canadian or Mexican who drove their car / truck across the border to go shopping at Target. Because of this, "analysts" usually equivocate and use terms like "consistent with" or "can't include / exclude." Which, again, is rhetoric - not science.