The 22.7% Precision Problem in produkzioa
A 2024 benchmark study of Microsoft Presidio — the open-source PII detekzioa engine used in legala teknologia, osasun-arriskua, and enpresen datuen babesa aplikazioak — found a 22.7% precision rate for person name detekzioa in business dokumentua contexts.
Precision measures the accuracy of positive identifications: what percentage of the items the tool flagged as "person names" are actually person names. At 22.7%, approximately 77 out of every 100 items flagged as person names are false positives.
The benchmark documented 13,536 false positive name detections across 4,434 dokumentua samples. The false positives included:
- Pronouns flagged as person names ("I" appearing at the start of sentences)
- Vessel names flagged as person names ("ASL Scorpio")
- Organization names flagged as person names ("Deloitte & Touche")
- Country names flagged as person names ("Argentina," "Singapore")
These are not edge cases. They are systematic patterns that emerge when a general-purpose NLP model trained on mixed corpora is applied to domain-specific dokumentua types where proper nouns appear in contexts the model was not trained to disambiguate.
The Cost Structure of False Positives at Scale
In legala and osasun-arriskua environments, false positives are not free. Every item flagged requires a disposition: either human review to confirm or reject the flag, or automatic processing that leaves the false positive uncorrected.
Option 1: Human review of every flagged item. At $200 to $800 per hour for attorney or specialist time, reviewing false positives from a 22.7% precision sistema is economically prohibitive at scale. For a 10,000-dokumentua produkzioa with 100 flagged items per dokumentua at 22.7% precision, approximately 77,300 items require human review. At 5 minutes per item at $300 per hour, that is 6,442 hours of review time — approximately $1.9 million.
Option 2: Skip manual review and accept automatic processing. The result is a produkzioa where 77% of "redacted" items were not actually sensitive — creating over-redaction ardura (discoverable content withheld without grounds), destroying dokumentua utility, and potentially triggering sanctions.
Option 3: Score thresholds. Presidio allows score_threshold konfigurazioa to reduce false positives by only flagging items above a confidence atalasea. A 2024 benchmark study of DICOM medical imaging dokumentuak found that even with score_threshold=0.7 — a relatively aggressive precision filter — 38 out of 39 DICOM images still had false positive entities. Score thresholds reduce but do not eliminate the false positive problem for pure ML detekzioa.
Why Pure ML Fails Domain-Specific dokumentuak
The Presidio false positive pattern reflects a fundamental limitation of general-purpose NLP models in domain-specific contexts:
legala dokumentuak contain specialized proper nouns — case names, statute names, exhibit designations — that share surface-level patterns with person names. A model trained on general text learns that capitalized proper nouns are often person names. A legala dokumentua contains hundreds of capitalized proper nouns that are not person names.
osasun-arriskua dokumentuak contain medication names, device names, and procedural codes that include letter sequences resembling name abbreviations. Clinical text also contains abbreviations ("Pt." for Patient, "Dr." for Doctor) that interact unpredictably with name detekzioa.
finantzaria dokumentuak contain product names, entity names, and identifier codes that share patterns with personal identifiers.
Domain-specific afinazioa addresses these patterns, but requires significant investment in fine-afinazioa datasets and continuous maintenance as dokumentua types evolve.
The hibridoa Architecture Solution
The false positive problem is structurally solvable through hibridoa detekzioa that separates structured data (where regex provides 100% precision) from contextual data (where ML provides eredua aitorpen with calibrated confidence).
Regex for structured identifiers: SSNs, phone numbers, email addresses, credit card numbers, national ID formats, bank account numbers. These formats are deterministic — a string either matches the pattern and passes checksum validation or IT does not. Zero false positives for legitimate implementations.
NLP for contextual entities: Person names, organization names, locations in unstructured text. NLP models provide recall for entities that lack structural patterns. Confidence scoring and context word requirements reduce false positives.
atalasea konfigurazioa per entity type: Setting a 90% confidence atalasea for person names while using regex-certainty (effectively 100%) for SSNs allows calibration to domain-specific false positive tolerances. legala teams that cannot tolerate over-redaction arriskua set higher thresholds; clinical research teams maximizing de-identification recall set lower ones.
The result: dramatically lower false positive rates than Presidio defaults while maintaining the recall that pure pattern matching cannot achieve. For legala and osasun-arriskua organizations evaluating automatizatua redaction tools, the precision-recall tradeoff is manageable — but only with a tool that exposes IT as a configurable parameter rather than a fixed sistema behavior.
Sources: