Ang Document Format Fragmentation Problem...

Ang Kalakasang Sitwasyon: Format-Specific Detection

Karamihan ng PII detection tools ay nag-optimize para sa isang format:

• Presidio → plain text / JSON logs
• OCR-based → scanned images + PDFs
• Regex scanners → CSV / database dumps
• Keyword matching → document corpuses

Sa production environments, data ay umabot mula sa 6+ formats parallel:

1. PDFs — contracts, forms, reports, scanned documents
2. Word documents (.docx) — memos, drafts, templates
3. Spreadsheets (.xlsx) — datasets, reporting, budgets
4. Screenshots (.png, .jpg) — chat transcripts, internal tool captures
5. Databases — patient records, customer data, logs
6. Structured data — JSON, XML, CSV, SQL dumps

Each format ay may sariling extraction mechanics at PII visibility pitfalls.

Format 1: PDFs

Problem: PDFs ay may dalawang categories:

• Text-layer PDFs — may underlying extractable text
• Image-based PDFs — scanned documents na page images

Ang generic text scanner ay nakikita ang text layer pero hindi ang scanned images.

Detection failure: Ang legal document ay 80% text-layer, 20% scanned. Ang generic scanner ay nag-flag ng text-layer content pero nag-miss ng SSN sa scanned section.

Solution: Multi-stage pipeline:

1. Extract text layer
2. Detect scanned images
3. Run OCR sa scanned images
4. Merge results
5. Deduplicate

Format 2: Word Documents

Problem: DOCX ay XML-based. Data ay maaaring maging:

• Plain text sa body
• Embedded textboxes
• Headers/footers
• Comments (tracked changes)
• Embedded images
• Embedded OLE objects

Maraming tools ay nag-extract ng body text lang.

Detection failure: Ang Word document ay may tracked changes na may sensitive salary data. Ang tool ay nag-scan ng visible text pero ang tracked changes ay hidden.

Solution:

1. Parse DOCX XML structure
2. Extract lahat ng text streams
3. Preserve metadata
4. Report PII sa non-body streams

Format 3: Spreadsheets

Problem: XLSX ay may multiple sheets + hidden content:

• Visible data sa sheets
• Hidden rows/columns
• Hidden sheets
• Embedded formulas
• Metadata

Detection failure: Ang HR spreadsheet ay may employee data sa Sheet1. Sheet2 ay hidden at may salary + SSN. Ang generic tool ay nag-scan ng Sheet1 lang.

Solution:

1. Enumerate lahat ng sheets
2. Expand hidden rows/columns
3. Evaluate formulas
4. Check metadata
5. Report sheet coverage

Format 4: Screenshots

Problem: Screenshots ay may:

• Rendered UI text (inconsistent quality)
• Form fields
• Blurred/pixelated content
• Multiple languages
• Variable resolution

Detection failure: Ang screenshot ay may email + phone. Ang OCR ay nag-extract ng text pero 60% lang ang accuracy.

Solution:

1. Preprocessing — enhance contrast, resize
2. Layout analysis — identify form fields
3. Multi-language OCR
4. Fuzzy matching
5. Confidence scoring

Format 5: Databases

Problem: Database PII ay hindi static:

• Data ay queried at runtime
• PII ay derived sa stored procedures
• Data ay decrypted on SELECT

Detection failure: Ang database ay may encrypted SSN. Ang tool ay nag-scan ng schema, mag-skip. Runtime query ay nag-decrypt pero ay hindi scanned.

Solution:

1. Query logging — capture queries + results
2. Runtime result scanning
3. Stored procedure analysis
4. Connection-time detection
5. Compliance note

Format 6: Structured Data

Problem: CSV/JSON/XML ay may variance sa encoding:

• CSV: quoted fields, escaped quotes
• JSON: unicode escapes, emoji, null bytes
• XML: CDATA sections, entities

Detection failure: Ang CSV ay may newline embedded sa quote. Ang tool ay mag-read bilang 3 lines, hindi 2.

Solution:

1. Format-aware parsing — use proper libraries
2. Unescape bago scan
3. Multi-line field handling
4. Encoding detection
5. Verify parse integrity

Building Format-Resilient Detection

Best practices:

1. Modular pipeline — separate extractors per format
2. Deduplication — same PII = single alert
3. Coverage tracking — report coverage %
4. Confidence tiers — regex vs. OCR matches
5. Fallback chains — primary then secondary extractor
6. Audit logging

Conclusion

Format fragmentation ay hindi edge case. Ito ay ang default state ng production data. Organizations na nag-rely sa single-format detection ay nakapalagay na compliant pero sa reality ay nag-scan lang ng portion ng data.