title: "Presidio False Positives: What They Cost in Legal and Healthcare" description: "A 2024 benchmark found Presidio generated 13,536 false positive name detections across 4,434 samples — flagging pronouns, vessel names, and countries as person names. Here is what that costs in legal and healthcare environments." category: technical publishedAt: 2026-03-23 tags:
- Presidio false positive rate
- PII detection precision
- automated redaction cost
- legal document review
- hybrid PII detection readingTime: 8
Updated for 2026
The 22.7% Precision Problem
A 2024 study tested Microsoft Presidio on business files. Presidio is an open-source PII tool. Legal teams and health groups use it widely.
The study measured how often Presidio was right. Of all the items it flagged as person names, how many were actually person names?
The answer was 22.7%. About 77 out of every 100 flags were wrong. The study counted 13,536 false flags across 4,434 sample files.
The errors were not random. They followed clear patterns:
- Pronouns flagged as people ("I" at the start of a sentence)
- Ship labels flagged as people ("ASL Scorpio")
- Company labels flagged as people ("Deloitte & Touche")
- Country terms flagged as people ("Argentina," "Singapore")
None of these are rare edge cases. They show up whenever a general NLP model meets domain-specific text. The model was not built to tell them apart.
What False Flags Cost
In legal and health work, every flag needs a response. Teams face three options. All three have real costs.
Option 1: A human checks every flag. Lawyer and expert time runs $200 to $800 per hour. At 22.7% accuracy, the volume is huge. This is not viable at scale. See eDiscovery PII Automation and Legal Review Cost Reduction for how review costs grow with volume.
Option 2: Skip review and trust the output. This is also risky. When 77% of "redacted" items are not sensitive, you create legal risk. Courts have fined lawyers for over-redaction. See eDiscovery Over-Redaction Sanctions for documented cases.
Option 3: Raise the score threshold. Presidio lets users set a score_threshold to drop weak flags. A 2024 DICOM study tested this at 0.7 — a fairly high bar. The result: 38 out of 39 DICOM images still had false flags. Thresholds help. They do not fix the root cause.
Why General NLP Struggles Here
The Presidio gap comes from a mismatch between training data and real-world use.
Legal files are full of capital-letter terms. Case names, law titles, and exhibit codes all look like personal data to a general model. It flags them. Most are not personal data.
Health files add drug names, device codes, and clinical short forms. "Pt." means Patient. "Dr." means Doctor. These trip up entity detection in ways that are hard to predict.
Finance files have product codes, entity strings, and account IDs that share surface patterns with personal records.
Fine-tuning a model on domain data helps. But it takes time and effort to build and keep up to date.
How Hybrid Detection Fixes This
The false flag problem has a clear fix. Split the work by data type.
Pattern rules for structured data. Social security numbers, phone numbers, email addresses, and ID formats follow fixed rules. A string either fits the pattern and passes a check digit test, or it does not. Zero false flags for valid rule sets.
Language models for free text. First and last names, company labels, and locations in prose lack rigid structure. NLP finds them when rules cannot. Confidence scores and context checks cut the false flag rate.
Per-type score settings for fine control. Legal teams that cannot risk over-redaction set high thresholds for fuzzy matches. Research teams that need high recall set lower ones. See Binary PII Detection and Confidence Scoring for Compliance for how score tiers work in practice.
The result is far fewer errors than Presidio defaults. Recall stays strong where rules alone would miss too much.
For legal and health teams, the key question is not whether false flags exist. They always do in NLP systems. The question is whether the tool lets you set, measure, and document the tradeoff.
When This Approach Has Limits
A hybrid design with tunable thresholds cuts the false-positive problem that plagues Presidio defaults, but it does not abolish the underlying tradeoff. Three caveats are worth stating plainly.
Precision and recall move against each other. Raising thresholds to stop the system flagging pronouns and country names as PII also raises the chance it misses a genuine name written in an unusual way. You are not eliminating error; you are choosing which kind you can tolerate. Legal teams tuning away over-redaction must accept a higher risk of an undetected identifier, and that is the more dangerous failure in a privacy context. The tool makes the tradeoff adjustable and measurable — it does not make it disappear.
Tuning requires labeled data and ongoing measurement. Setting thresholds well depends on a representative, accurately labeled sample of your own documents and a process to re-measure as document types change. Teams without that ground truth are guessing at threshold values, and a setting tuned for one corpus can perform poorly on another. The capability to tune is only as good as the evaluation discipline behind it.
Lower false positives still leave residual false negatives. Even a well-tuned hybrid system misses some PII, particularly in OCR-degraded scans, mixed-language text, or formats outside its recognizers. For legal and healthcare releases, that residual rate is why human review of the output remains necessary. The system reduces the review burden; it does not remove the obligation to verify before sensitive material leaves the firm.