LLM Privacy Attack Research
12 peer-reviewed research papers demonstrating why pseudonymity fails against AI.
Deanonymization, PII extraction, membership inference, prompt injection attacks — and how to protect against them.
Privacy Attack Categories
Deanonymization
LLMs match anonymous posts to real identities using writing style, facts, and temporal patterns. 68% accuracy at $1-$4/profile.
Attribute Inference
LLMs infer personal attributes (location, income, age) from text even when not stated. GPT-4 achieves 85% top-1 accuracy.
PII Extraction
Extracting personal information from training data or prompts. 100% email extraction accuracy with GPT-4. 5× increase with advanced attacks.
Prompt Injection
Manipulating LLM agents to leak personal data during task execution. ~20% attack success rate on banking scenarios.
Large-scale online deanonymization with LLMs
Simon Lermen (MATS), Daniel Paleka (ETH Zurich), Joshua Swanson (ETH Zurich), Michael Aerni (ETH Zurich), Nicholas Carlini (Anthropic), Florian Tramèr (ETH Zurich)
Published: February 18, 2026
Key Finding
68% recall at 90% precision for deanonymization using ESRC framework
Methodology
Designed attacks for closed-world setting with scalable attack pipeline using LLMs to: (1) extract identity-relevant features, (2) search for candidate matches via semantic embeddings, (3) reason over top candidates to verify matches and reduce false positives.
ESRC Framework
LLM extracts identifying facts from anonymous posts
Uses facts to query public databases (LinkedIn, etc.)
LLM reasons about candidate matches
Confidence scoring to minimize false positives
Experimental Results
| Dataset | Recall @ 90% Precision | Notes |
|---|---|---|
| Hacker News → LinkedIn | 68% | vs near 0% for classical methods |
| Reddit cross-community | 8.5% | Multiple subreddits |
| Reddit temporal split | 67% | Same user over time |
| Internet-scale (extrapolated) | 35% | At 1M candidates |
Implications
Practical obscurity protecting pseudonymous users online no longer holds. Classical methods achieve near 0% recall under same conditions.
All Research Papers
11 additional peer-reviewed studies on LLM privacy attacks
Beyond Memorization: Violating Privacy via Inference with Large Language Models
Robin Staab, Mark Vero, Mislav Balunović, et al. (ETH Zurich)
85% top-1 accuracy inferring personal attributes from Reddit posts
First comprehensive study on LLM capabilities to infer personal attributes from text. GPT-4 achieved highest accuracy among 9 tested models.
Key Findings
- •85% top-1 accuracy, 95% top-3 accuracy at inferring personal attributes
- •100× cheaper and 240× faster than human annotators
- •Tested 9 state-of-the-art LLMs including GPT-4, Claude 2, Llama 2
- •Infers location, income, age, sex, profession from subtle text cues
AutoProfiler: Automated Profile Inference with Language Model Agents
Yuntao Du, Zitao Li, Bolin Ding, et al. (Virginia Tech, Alibaba, Purdue University)
85-92% accuracy for automated profiling at scale using four specialized LLM agents
Framework using specialized LLM agents (Strategist, Extractor, Retriever, Summarizer) for automated profile inference from pseudonymous platforms.
Key Findings
- •Four specialized agents: Strategist, Extractor, Retriever, Summarizer
- •Iterative workflow enables sequential scraping, analysis, and inference
- •Outperforms baseline FTI across all attributes and LLM backbones
- •Short-term memory for Extractor/Retriever, long-term memory for Strategist/Summarizer
Large Language Models are Advanced Anonymizers
Robin Staab, Mark Vero, Mislav Balunović, et al. (ETH Zurich SRI Lab)
Adversarial anonymization reduces attribute inference from 66.3% to 45.3% after 3 iterations
LLMs can be used defensively in adversarial framework to anonymize text. Outperforms commercial anonymizers in both privacy and utility.
Key Findings
- •Adversarial feedback enables anonymization of significantly finer details
- •Attribute inference accuracy drops from 66.3% to 45.3% after 3 iterations
- •Evaluated 13 LLMs on real-world and synthetic online texts
- •Human study (n=50) showed strong preference for LLM-anonymized texts
AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents
Arman Zharmagambetov, Chuan Guo, Ivan Evtimov, et al. (Meta AI, CMU)
GPT-4, Llama-3, and Claude web agents are prone to inadvertent use of unnecessary sensitive information
Benchmark measuring if AI web agents follow data minimization principle. Simulates realistic web interactions across GitLab, Shopping, and Reddit.
Key Findings
- •Evaluates GPT-4, Llama-3, Claude-powered web navigation agents
- •Measures data minimization compliance: use PII only if 'necessary' for task
- •Agents often leak sensitive information when unnecessary
- •Three test environments: GitLab, Shopping, Reddit web apps
SoK: The Privacy Paradox in Large Language Models
Various researchers
Systematization of 5 distinct privacy incident categories beyond memorization
Comprehensive survey categorizing privacy risks: training data leakage, chat leakage, context leakage, attribute inference, and attribute aggregation.
Key Findings
- •Five privacy incident categories identified:
- •1. Training data leakage via regurgitation
- •2. Direct chat leakage through provider breaches
- •3. Indirect context leakage via agents and prompt injection
PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMs
Krishna Kanth Nakka, Ahmed Frikha, Ricardo Mendes, et al. (Various)
PII extraction rates increase up to 5× with sophisticated adversarial capabilities and limited query budget
Comprehensive benchmark for PII extraction attacks. Reveals notable underestimation of PII leakage in existing single-query attacks.
Key Findings
- •PII extraction rates can increase up to 5× with sophisticated attacks
- •Existing single-query attacks notably underestimate PII leakage
- •Taxonomy: Black-box (True-prefix, ICL, PII Compass) and White-box (SPT) attacks
- •Hyperparameters like demonstration selection crucial to attack effectiveness
Evaluating LLM-based Personal Information Extraction and Countermeasures
Yupei Liu, Yuqi Jia, Jinyuan Jia, et al. (Penn State, Duke University)
GPT-4 achieves 100% accuracy extracting emails and 98% for phone numbers from synthetic profiles
Systematic measurement study benchmarking LLM-based personal information extraction (PIE). Proposes prompt injection as novel defense.
Key Findings
- •GPT-4: 100% email extraction, 98% phone number extraction on synthetic data
- •Larger LLMs more successful: vicuna-7b achieves 65%/95% vs GPT-4's 100%/98%
- •LLMs better at: emails, phone numbers, addresses, names
- •LLMs worse at: work experience, education, affiliation, occupation
Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
Michele Miranda, Elena Sofia Ruzzetti, Andrea Santilli, et al. (Various)
Comprehensive taxonomy of privacy attacks: training data extraction, membership inference, model inversion
Survey examining privacy threats from LLM memorization. Proposes solutions from dataset anonymization to differential privacy and machine unlearning.
Key Findings
- •Privacy attacks covered: Training data extraction, Membership inference, Model inversion
- •Training data extraction: non-adversarial and adversarial prompting
- •Membership inference: shadow models and threshold-based approaches
- •Model inversion: output inversion and gradient inversion
Beyond Data Privacy: New Privacy Risks for Large Language Models
Various researchers
LLM autonomous capabilities create new vulnerabilities for inadvertent data leakage and malicious exfiltration
Explores privacy vulnerabilities from LLM integration into applications and weaponization of autonomous abilities.
Key Findings
- •LLM integration creates new privacy vulnerabilities beyond traditional risks
- •Opportunities for both inadvertent leakage and malicious exfiltration
- •Adversaries can exploit systems for sophisticated large-scale privacy attacks
- •Autonomous LLM abilities can be weaponized for data exfiltration
Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents
Various researchers
15-50% utility drop under attack with ~20% average attack success rate for personal data leakage
Examines prompt injection causing tool-calling agents to leak personal data during task execution. Uses fictitious banking agent scenario.
Key Findings
- •16 user tasks from AgentDojo benchmark evaluated
- •15-50 percentage point drop in LLM utility under attack
- •~20% average attack success rate across LLMs
- •Most LLMs avoid leaking passwords due to safety alignments
Membership Inference Attacks on Large-Scale Models: A Survey
Various researchers
First comprehensive review of MIAs targeting LLMs and LMMs across pre-training, fine-tuning, alignment, and RAG stages
Survey analyzing membership inference attacks by model type, adversarial knowledge, strategy, and pipeline stage.
Key Findings
- •Analyzes MIAs across: pre-training, fine-tuning, alignment, RAG stages
- •Strong MIAs require training multiple reference models (computationally expensive)
- •Weaker attacks often perform no better than random guessing
- •Tokenizers identified as new attack vector for membership inference
Defense Strategies from Research
What Doesn't Work
- ✗Pseudonymization — LLMs defeat usernames, handles, display names
- ✗Text-to-image conversion — Only slight decrease against multimodal LLMs
- ✗Model alignment alone — Currently ineffective at preventing inference
- ✗Simple text anonymization — Insufficient against LLM reasoning
What Does Work
- ✓Adversarial anonymization — Reduces inference 66.3% → 45.3%
- ✓Differential privacy — Reduces PII precision 33.86% → 9.37%
- ✓Prompt injection defense — Most effective against LLM-based PIE
- ✓True PII removal/replacement — Removes signals LLMs use
Why This Research Matters
These 12 research papers demonstrate a fundamental shift in privacy threats. Traditional anonymization approaches like pseudonyms, usernames, and handle changes are no longer sufficient protection against determined adversaries with access to LLMs.
Key Threat Metrics
- 68% deanonymization accuracy at 90% precision (Hacker News → LinkedIn)
- 85% attribute inference accuracy for location, income, age, occupation
- 100% email extraction and 98% phone number extraction (GPT-4)
- 5× increase in PII leakage with sophisticated multi-query attacks
- $1-$4 cost per profile makes mass attacks economically feasible
Who Is At Risk
- Whistleblowers & activists: Anonymous posts can be linked to real identities
- Professionals: Reddit activity linked to LinkedIn profiles
- Healthcare patients: Membership inference reveals if data was in training
- Anyone with historical posts: Years of data can be retroactively deanonymized
How anonym.legal Addresses These Threats
anonym.legal provides true anonymization that removes the signals LLMs use:
- 285+ Entity Types: Names, locations, dates, temporal markers, identifiers
- Writing Pattern Disruption: Replaces text that reveals stylometric fingerprints
- Reversible Encryption: AES-256-GCM for cases requiring authorized access
- Multiple Operators: Replace, Redact, Hash, Encrypt, Mask, Custom
Frequently Asked Questions
What is LLM-based deanonymization?
LLM-based deanonymization uses large language models to identify real individuals from anonymous or pseudonymous online posts. Unlike traditional methods that fail at scale, LLMs can combine writing style analysis (stylometry), stated facts, temporal patterns, and contextual reasoning to match anonymous profiles to real identities. Research shows up to 68% accuracy at 90% precision, compared to near 0% for classical methods.
How accurate is LLM deanonymization?
Research demonstrates alarming accuracy levels: 68% recall at 90% precision for Hacker News to LinkedIn matching, 67% for Reddit temporal analysis (same user over time), 35% at internet-scale (1M+ candidates). For attribute inference, GPT-4 achieves 85% top-1 accuracy inferring personal attributes like location, income, age, and occupation from Reddit posts alone.
What is the ESRC framework?
ESRC (Extract-Search-Reason-Calibrate) is a four-step LLM deanonymization framework: (1) Extract - LLM extracts identifying facts from anonymous posts using NLP, (2) Search - queries public databases like LinkedIn using extracted facts and semantic embeddings, (3) Reason - LLM reasons about candidate matches analyzing consistency, (4) Calibrate - confidence scoring to minimize false positives while maximizing true matches.
How much does LLM deanonymization cost?
Research shows LLM-based deanonymization costs $1-$4 per profile, making mass deanonymization economically feasible. For defensive anonymization, costs are under $0.035 per comment using GPT-4. This low cost enables state actors, corporations, stalkers, and malicious individuals to perform large-scale privacy attacks.
What types of PII can LLMs extract from text?
LLMs excel at extracting: email addresses (100% accuracy with GPT-4), phone numbers (98%), mailing addresses, and names. They can also infer non-explicit PII: location, income level, age, sex, occupation, education, relationship status, and place of birth from subtle textual cues and writing patterns.
What is a membership inference attack (MIA)?
Membership inference attacks determine whether specific data was used to train an AI model. For LLMs, this reveals if your personal information was in the training dataset. Research shows email addresses and phone numbers are particularly vulnerable. New attack vectors include tokenizer-based inference and attention signal analysis (AttenMIA).
How do prompt injection attacks leak personal data?
Prompt injection manipulates LLM agents to leak personal data observed during task execution. In banking agent scenarios, attacks achieve ~20% success rate at exfiltrating personal data, with 15-50% utility degradation under attack. While safety alignments prevent password leakage, other personal data remains vulnerable.
How can anonym.legal help protect against LLM privacy attacks?
anonym.legal provides true anonymization through: (1) PII Detection - 285+ entity types including names, locations, dates, writing patterns, (2) Replacement - substitutes real PII with format-valid alternatives, (3) Redaction - completely removes sensitive information, (4) Reversible Encryption - AES-256-GCM for authorized access. Unlike pseudonymization which LLMs defeat, true anonymization removes the signals LLMs use for deanonymization.
Protect Against LLM Privacy Attacks
Don't rely on pseudonymity. Use true anonymization to protect sensitive documents, user data, and communications from AI-powered identification attacks.