Here's the "so what" answer. Let's examine the business of DVR manufacturing from the standpoint of determining the DVR's purpose and if it fulfills its purpose. Attempting to identify purpose / fit for purpose of the parts in the recording chain is one of the elements of the Content Triage step in the processing workflow. Why did the device produce a recording of five white pixels in the area where you were expecting to see a license plate? Understanding purpose helps answer these "why" questions.
What is the purpose of a generic Chinese 4 channel DVR? The answer is not what you think.
For our test, we'll examine a generic Chinese 4 channel DVR, the kind found at any convenience store around the US. It captured a video of a crime and now you want to use it's footage to answer questions about the events of that day. Can you trust it?
Take a DVR sold on Amazon or any big box retailer. There's the retail price, and there's the mark-up along the way to the retailer.
When you drill down through the distribution chain to the manufacturer, you find out something quite amazing, like this from Alibaba.com.
The average wholesale price of a 4 channel DVR made in China is $30 / unit. Units with more camera channels aren't much more. Units without megapixel recording capability are a bit less. This price is offered with the manufacturer's profit built in. Given that the wholesale price includes a minimum of 100% markup from cost, and that there is a labor and fixed costs involved, the average Chinese DVR is simply a $7 box of parts. The composition of that box of parts is entirely dependent upon what's in the supply chain on the day the manufacturing order was placed. That day's run may feature encoding chips from multiple manufacturers, as an example. The manufacturer does not know which unit has chips from a particular manufacture - and doesn't care as long as it "works."
What's the purpose of this DVR? The purpose has nothing to do with recording your event. The purpose is to make about $15 in profit for the manufacturer whilst spending about $15 on parts, labor, and overhead. Check again for 4 channel DVRs on Alibaba.com. There's more than 2500 different manufacturers in China offering a variety of specs within this space ... all making money with their $7 box of parts.
Let's say the $7 of parts at your crime scene recorded your event at 4CIF. You are asked to make some determination that involves time. You'll want to know if you can trust your $7 box of parts to accurately record time. How would you know?
One of the more popular DVR brands out west is Samsung. But, Samsung doesn't exist as such anymore. Samsung Techwin (Samsung's CCTV business unit) was sold to Hanwha Group a few years ago and is now sold as Hanwha Techwin (Samsung Techwin) in the US. Where does Hanwha get their $7's worth of parts within the supply chain? China, for the most part. China can make DVR parts a lot cheaper than their Korean counterparts.
Here's the specs from a Hanwha Techwin HRD-440.
This model, recording at 4CIF, for example, can record UP TO 120fps across all of it's channels. UP TO means it's max potential recording rate. It does not mean it's ACTUAL recording rate at the time of the event in question. The "up to" language is placed there to protect the manufacturer of this $7 box of parts against performance claims. If it was a Swiss chronometer, it wouldn't need the disclaiming language. But, it's not a Swiss chronometer - it's a $7 box of parts.
What does the recording performance of the channel in question in the specific evidentiary DVR look like when it alone is under load (maximum potential recording rate)? What about the recording performance of the channel in question (at max) when the other channels move in and out of their own maximum potential recording rate? What happens within the system when all channels are at the max? Remember also that systems like these allow for non-event recording to happen at lower resolutions than event recording (alarm / motion). How does the system respond when a channel or all channels are switching resolutions up / down? How does what's happening internally compare with the files that are output to .avi or .sec files? How do these compare to data that's retrieved and processed via direct acquisition of the hard drive?
How would you know? You would build a performance model. That's something that you learn in all the stats / quant classes that you take along the way to earning a PhD. I earned my PhD in Education.
Why a PhD in Education, you might ask. Three reasons. There are no PhDs in Forensic Multimedia Analysis for one. The second reason, and the subject of my dissertation, deals with the environment on campus and in the classroom that causes such a great number of otherwise well qualified people to arrive on campus and suddenly and voluntarily quit (withdraw). The results of my research can be applied to help colleges configure their classes and their curriculum, as well as to train professors to accommodate a diverse range of students - including mature adults with a wealth of knowledge who arrive in class with fully formed and sincerely held opinions. The third reason has to do with a charity that I founded a few years ago to help bring STEM educational help to an underserved community and population of learners in the mountain communities of northern Los Angeles / southern Kern counties in California.
Imagine that you've been told by your chain of command that you must have certain level of education to promote at your agency. That's what happened to me. I was minding my own business with a AS in Political Science that I cobbled together after my college football career, such as it was, crashed and burned after injury. I later found myself in police service when these new rules were instituted. But, thankfully, our local Sheriff had approached the local schools promising butts in seats if they'd only reduce their tuition. So I finished my Bachelors degree at an esteemed B-school for $7k and stayed there for a MOL for only $9k. The PhD path wasn't cheap, but it was significantly cheaper than it would have been without the Sheriff's office's help. As to why I chose to go all the way to PhD, that was the level of education necessary to make more pensionable money had I decided to switch from being a technician making more than half-again my salary in overtime (which isn't pensionable, sadly) to management. But, I digress. Back to work, Jim.
Sparing you the lecture on time and temporality here, the basic tenet of experimental science is that you can only measure "now." If you want to know what happened / will happen, you need to build a model. Meteorologists build a model of future environmental patterns to forecast the weather for next week. They don't measure next week's weather properties today. The same hold true across the sciences. Moneyball was a Quant's attempt to model behavior in order to achieve a future advantage in sports.
When modeling performance, it's important to use valid tools and to control for all variables (as best as possible). At a minimum, it's important to know how your tools are working and how to not only interpret the results produced but to spot issues of concern within the results.
As an example, pretty much everyone in this space is familiar with FFMPEG and it's various parts. Let's say that you use the command line version to analyze the stream and container of the .avi file from our example DVR (it's all you have to work with). It's an NTSC DVR and the results from your analysis tool indicate a frames per second (fps) of 25. Is this correct? Would you necessarily expect 25fps from an NTSC DVR? Is this FFMPEG's default when there's no fps information in the file (it's a European tool after all)? Does total frames / time = 25fps? If yes, you're fine. If not, what do you do? You test.
Is your single evidentiary file (sample size = 1) sufficient to generalize the performance of your $7 box of parts? Of course not. In order to know how many samples are needed to generalize the results across the population of files from this specific DVR, you need to test - to build a performance model. How many unique tests will gain you the appropriate number of samples from which to build your model? Well, that depends on the question, the variables, and the analysts' tolerance for error ...
