Running 50K Clinical Notes Locally: HIPAA Guide
Research teams that need to de-identify large note archives face a common gap. Cloud tools often can't handle the volume. Many rules require on-site work. Manual review takes too long. Local batch runs are the answer.
This guide covers the key rules, the setup, and the records you need.
See our compliance overview and security practices for how we support HIPAA.
Why Cloud Does Not Work Here
HIPAA's Expert Determination method sets a clear bar. De-identified data must carry "very small risk" of re-identification. A qualified person must verify that. An IRB that approves research with de-identified patient data also needs records. You must document the method used, the entity types removed, and the quality checks applied.
That records requirement is key. De-identification can't be a black box. You must show what was found, what was removed, and how you checked the result.
Uploading 500,000 files to a cloud API is slow and costly. Rate limits and long transfer times make it hard. Cloud runs are rarely practical for large research datasets.
HIPAA adds a second concern. Sending protected health information (PHI) to a Business Associate — even a de-identification vendor — requires a Business Associate Agreement (BAA). For IRB research, BAA rules may intersect with IRB data use terms. Legal review is often needed. Local runs remove the data-transfer concern entirely.
Why the Privilege Case Matters
A February 2026 SDNY ruling found that AI-processed documents lose attorney-client privilege if not anonymized first. The court held that sending privileged documents to an external AI service was a disclosure. That disclosure waived privilege for the content analyzed.
The healthcare parallel is clear. Physician notes sent to cloud NLP tools carry similar risk. Therapist records sent to outside AI services do too. Local runs — where documents never leave your site — avoid that risk.
See our guide on HIPAA cloud and zero-knowledge PHI for more on keeping data on-site.
How to Set Up for 50K Notes
Batch size: The Desktop App handles 1–5,000 files per batch based on your plan. Ten batches of 5,000 covers all 50,000 notes in one overnight job. No manual steps are needed in between.
Speed: Running 1–5 files at once boosts output. A single overnight job finishes the full set with no extra work.
Entity types: Healthcare-specific types include MRN formats, NPI numbers, DEA numbers, health plan IDs, and HIPAA date formats. Set them once in a named preset. That preset applies to every batch. De-identification stays uniform across all files.
Audit logs: Each batch job exports a CSV or JSON file. It records the file name, entity types found, confidence scores, and a time stamp. This log meets the IRB Expert Determination requirement. You can show what was found and removed in each file.
IRB Records Checklist
Before you file your IRB protocol, confirm you can show:
- Name and version of the de-identification tool
- Full list of entity types in the preset
- Test results on a held-out sample
- Batch logs for each run (file name, entity counts, time stamp)
- Proof that no PHI left your on-site environment
Local batch runs make each item easy to produce. Logs are auto-generated. The preset is saved and versioned. The site boundary is clear.
When This Approach Has Limits
Local batch de-identification gives you a clean site boundary and the audit records IRB and Expert Determination require. But producing the evidence is not the same as satisfying the standard, and three limits apply.
Expert Determination is a statistician's judgment, not a batch output. HIPAA's Expert Determination method requires a qualified expert to certify that re-identification risk is very small for the specific dataset, considering quasi-identifiers and the data's context. Versioned presets, held-out test results, and batch logs are inputs the expert reviews; they do not replace the determination itself. A clean run does not make a dataset certified — a person with the right expertise still has to sign off.
Batch scale magnifies a systematic miss. Running 50,000 notes through one preset means any gap in that preset repeats 50,000 times. A note type, abbreviation, or free-text pattern the configuration does not handle leaves residual PHI across the whole corpus uniformly. Held-out sample testing catches systematic errors only if the sample represents the full diversity of the notes — validate against the messiest and least common formats, not just typical ones.
Local processing controls custody, not detection accuracy. Keeping PHI on-site eliminates cloud transfer risk and the BAA question, which is a genuine advantage. It does nothing for the residual false-negative rate inherent in automated detection. The strongest workflow still includes human review of a meaningful sample before a research dataset is released, because the irreversible failure in de-identification is the identifier that was missed, not the one that was logged.