False Conviction

In one sentence

The "inconclusive" loophole in forensic validation studies allows experts to claim near-perfect accuracy while masking a much higher potential error rate in real-world criminal cases.

Why read this (and who it's for)

This piece is for anyone who values evidence-based truth—lawyers, students, jurors, or anyone interested in how "expert" testimony can be manipulated. It solves the problem of blind trust in "expert" statistics by teaching you how to identify when a number is being used to obscure a reality. After reading this, you will be able to look at "accuracy" claims and ask the right questions about how "inconclusive" results were handled.

The big idea

The central thesis is that the validation of forensic disciplines (like fingerprint and firearms identification) is fundamentally flawed because of how "accuracy" is scored. In many studies, forensic experts are asked to identify matches; if they are unsure, they can answer "inconclusive." Under current scoring systems, "inconclusive" is treated as a correct answer or is dropped entirely.

This creates a "False Conviction." Because "inconclusive" is never counted as an error, an expert could theoretically achieve a 100% accuracy score by simply refusing to make a definitive call on any difficult case. This allows the forensic community to claim a 0% or 1% error rate in court, even though the actual ability of the expert to distinguish a match from a non-match may be significantly lower.

Key ideas

The "Inconclusive" Loophole

• What it is — In forensic validation tests, "inconclusive" answers are not counted as errors. This means an expert can avoid a "wrong" score by simply not providing a definitive answer.
• Why it matters — It allows for a "perfect" score that doesn't actually reflect the expert's ability to identify a match. It creates a false sense of certainty for juries who hear that a discipline is "accurate."
• In the book — The author notes that if an expert identifies one match correctly and answers "inconclusive" for the other nine, they would receive a 100% score.

The Reputation Incentive

• What it is — Forensic examiners are often incentivized to be more "conclusive" in real-world casework than in validation studies.
• Why it matters — Because they know that an "incorrect" response in a validation study could "adversely affect the theory of firearms identification" in legal proceedings.
• In the book — Dror and Scurich noted that examiners often resort to more "inconclusive" decisions during studies to protect the reputation of their field, even though they are more conclusive when real lives are on the line.

The Calculation Gap

• What it is — The difference between a 1% error rate and a 35% error rate depends entirely on how you choose to count "inconclusive" results.
• Why it matters — It demonstrates how the same set of data can be used to tell two completely different stories about safety and reliability.
• In the book — In a study of 2,178 comparisons, the error rate was 1% if "inconclusive" was counted as correct, but it would be 35% if "inconclusive" was counted as an error.

The Discrepancy of Validation

• What it is — Validation studies are often performed under "blind" conditions where the expert knows they are being tested, which may not reflect the pressure of a real trial.
• Why it matters — If experts are incentivized to avoid "wrong" answers to protect their field's reputation, the "accuracy" they claim may be a byproduct of the testing environment rather than a reflection of scientific reality.
• In the book — The author highlights that "inconclusive" is often the only responsible answer in some cases, but the way those results are reported to the public and the court is what matters.

Frameworks & tools

The Accuracy Audit

Use this checklist when evaluating any "accuracy" statistic provided by an expert or a study:

1. Identify the "Accuracy" Claim: What is the reported error rate? (e.g., "98% accurate").
2. Identify the "Inconclusive" Count: How many of the total samples were labeled as "inconclusive"?
3. Recalculate the "True" Error Rate: Perform the calculation: (False Positives + Inconclusives) / Total Samples.
4. Compare the Results: Compare the reported error rate (which ignores inconclusives) against the recalculated rate (which treats inconclusives as a failure to identify).

Memorable examples & stories

The 2009 Brooklyn Case

David, the author, sat on a jury in a 2009 case where forensic experts testified that shell casings came from a specific gun. The defense didn't argue the possibility of a mismatch, and the jury never questioned the forensic testimony. This serves as a warning of how easily "expert" testimony is treated as gospel.

The 10-Item School Test

The author uses a simple analogy: If you took a 10-question test and were allowed to answer "inconclusive" for nine questions, you could achieve a 100% score without actually getting a single question right. This highlights the absurdity of current forensic scoring.

The 2,178 Comparison Study

In a study of 2,178 shell cartridge comparisons, experts were asked to identify non-matches. They identified 1,421 correctly, made 22 false-positive identifications, and left 735 as "inconclusive." By counting "inconclusive" as correct, the error rate was reported as 1%. If "inconclusive" were counted as an error, the error rate would be 35%.

Key quotes

"Inconclusive is never counted as incorrect." Why it matters: This is the mechanical flaw that allows for the creation of "false" accuracy.

"The potential error rate is much, much higher than the reported error rate." Why it matters: It warns the reader that the "official" numbers often hide the true risks of forensic science.

"The reality is, these validation studies are obscuring the truth about forensic science rather than revealing it." (paraphrased) Why it matters: It highlights that the current system of validation is producing the opposite of its intended effect.

How to apply it

This week

• Audit a Statistic: Find a "success rate" or "accuracy" claim in a news article or advertisement. Ask yourself: "How many of these were 'inconclusive' or 'not applicable'?"
• The "Inconclusive" Test: Next time you see a "99% success rate," mentally recalculate it by asking, "What would this number be if the 'inconclusive' cases were counted as failures?"

Ongoing

• Question the "Expert": When an expert provides a statistic, ask: "How were the inconclusive results handled in this study?"
• Demand Context: Always ask for the "denominator." If someone says a test is 90% accurate, ask what the total number of tests was and how many were deemed "inconclusive."

Common pitfalls

• Assuming "Inconclusive" is always wrong: The author clarifies that "inconclusive" may be the only responsible answer in some cases. The problem isn't the answer itself; it's how that answer is scored and reported.
• Trusting "Expert" status as a shield: Just because a forensic expert is an expert doesn't mean their reported accuracy is a reflection of their ability to identify a match; it may simply be a reflection of how they were allowed to score the test.

Connects to

• Thinking, Fast and Slow by Daniel Kahneman (specifically the sections on System 1's tendency to accept authority).
• The Art of Statistics by David Spiegel (for understanding how data can be manipulated to tell different stories).

If you remember only three things

• "Inconclusive" is often counted as a correct answer in forensic studies, which artificially inflates accuracy ratings.
• The difference between a 1% error rate and a 35% error rate can be as simple as how you choose to count "inconclusive" results.
• Experts are often incentivized to avoid "wrong" answers to protect the reputation of their field, which can lead to a "False Conviction" of certainty in the courtroom.

Internalize it

• Reflection questions

  • When have I accepted a "success rate" without asking what happened to the "inconclusive" cases?
  • How does the "reputation incentive" affect the way I present my own professional expertise?
  • If I were on trial, what specific information would I want to know about the "real" error rate of forensic evidence?

• Flashcards Q: How are "inconclusive" results currently scored in many forensic validation studies? A: They are often counted as correct answers or dropped entirely, meaning they are never counted as errors. Q: What was the difference in error rates in the study of 2,178 shell casing comparisons? A: The error rate was 1% if "inconclusive" was counted as correct, but 35% if "inconclusive" was counted as an error. Q: Why do forensic experts sometimes provide more "conclusive" answers in real cases than in validation studies? A: Because they are incentivized to avoid answers that might adversely affect the reputation of their field in legal proceedings. Q: What is the "False Conviction" described in the text? A: The false sense of certainty created when a jury hears that a forensic discipline is "100% accurate" because of flawed validation scoring. Q: Is an "inconclusive" answer always a wrong answer? A: No, it may be the only responsible answer in cases where the evidence is truly inconclusive.

Fidelity note

This summary is based strictly on the blog post "Firearms and Fingerprints: Forensic Accuracy Stats Don't Mean What They Say" by David Epstein. The "book" is a long-form blog post. I am highly confident in the technical breakdown of the "inconclusive" scoring system and the 1% vs 35% error rate calculation. The author's personal experience as a juror is included as a primary example of how these issues manifest in the real world.