By · Last updated 2026-02-26

返回博客技术

多语言NER:英语模型在阿拉伯语上的失败

英语NER模型准确率达85-92%。阿拉伯语和中文呢?通常只有50-70%。了解技术挑战以及如何构建真正的多语言保护。

February 26, 20268 分钟阅读
NERmultilingualArabic NLPChinese NLPPII detection

英语优先的NER问题

2026年更新

命名实体识别(NER)模型在英语文本上表现优异。在标准基准测试中,准确率达85-92%。这些数字具有误导性,特别是对于在多语言环境中运营的组织而言。

当同样的模型处理阿拉伯语或中文文本时,准确率通常降至50-70%。对于GDPR合规场景,这意味着一半到近一半的PII实例可能被遗漏。

技术根本原因

阿拉伯语的挑战

形态复杂性:阿拉伯语使用词根-词缀系统,一个词根可以生成数十种词形。人名"Mohammed"在真实文本中可能出现为"Al-Mohammed"、"bin Mohammed"或"Mohammed al-Rashid"。针对西方名字格式构建的正则表达式模式会遗漏这些变体。

右至左书写方向:RTL文本需要不同的分词和分段逻辑。实体边界在LTR和RTL文本中的工作方式不同。在英语上训练的NER系统应用LTR规则,这些规则在RTL文本上会产生错误的实体边界。

无连字符:阿拉伯语不像英语那样用空格分隔所有词语,这给基于标记的NER模型带来了额外挑战。

中文的挑战

无词语分隔符:中文不在词语之间使用空格。"王小明在北京工作"这个句子没有空格,但包含一个人名(王小明)和一个地名(北京)。分词是NER的前提步骤,若处理不当,实体识别将失败。

字符歧义:同一汉字在不同语境下可能是名字的一部分,也可能是普通词语。"明"可以是名字的一部分,也可以是"明亮"的意思。基于上下文的歧义消解需要专门训练。

繁简体差异:繁体字(台湾、香港)和简体字(中国大陆)存在不同的字符形式,英语优先模型通常未经专门训练无法处理这种差异。

越南语的挑战

越南语使用拉丁字母,但有声调标记(如"Nguyễn")。许多NER工具会去除这些标记后处理,导致无法区分同形的越南名字。

GDPR合规影响

GDPR不关心语言。第4(1)条以语言无关的方式定义"个人数据"。德国Steuer-ID、法国NIR号码或阿拉伯语人名的遗漏,与遗漏美国社会安全号码产生相同的法律风险。

这创造了一个可量化的合规缺口:

  • 为英语优化的工具在处理阿拉伯语文本时准确率为50-70%
  • 30-50%的遗漏率意味着同等比例的未受保护PII
  • 每个未受保护的标识符都代表着潜在的GDPR第32条违规

跨语言检测方法

有效的多语言PII检测需要多层架构:

语言识别层:首先检测文本语言,然后应用相应的模型和规则。

基于规则的检测:针对结构化标识符(ID号码、电话格式、IBAN)使用语言特定的正则表达式,包含校验和验证。

跨语言神经模型:XLM-RoBERTa等模型在100多种语言上预训练,优于单语言模型,尤其在训练数据较少的语言上表现更佳。

实体类型专业化:人名检测与ID号码检测需要不同的模型——前者是语言高度相关的,后者更多依赖结构性规则。

anonym.legal的检测覆盖48种语言,285种以上实体类型,专为多语言企业工作流设计。请参阅实体检测概述多语言PII指南了解详情。

参考资料

准备好保护您的数据了吗?

开始使用 285 种实体类型在 48 种语言中匿名化 PII。

About this page

We update this page when our platform or the law changes.

Read our founder note for how we work.

Each change shows up in the timestamp at the top.

Related reading

We follow these rules

  • GDPR (EU 2016/679).
  • ISO/IEC 27001:2022.
  • NIS2 (EU 2022/2555).
  • HIPAA safe harbor under 45 CFR § 164.514(b)(2).

Our promise

We do not sell your data.

We do not train models on your text.

We store your files in Germany.

You can delete your account at any time.

You own your work.

Where we run

Our servers live in Falkenstein, Germany.

We use Hetzner. They hold ISO 27001 certification.

All data stays in the EU.

Backups run every day.

Need help?

Email support@anonym.legal.

We reply within one business day.

How we test

We run a full check suite on every release.

Each surface gets its own sweep script and report.

Human reviewers spot-check the output each week.

We track recall and precision on a labelled set.

Bad runs block the deploy.

What we never do

  • We never sell your information to third parties.
  • We never train models on what you upload.
  • We never keep your work after you delete it.
  • We never share keys with any outside firm.
  • We never run ads inside the product.

Plans in plain words

We sell credits, not seats.

One credit covers one short job.

Long jobs use a few credits each.

You can top up at any time.

Unused credits roll over each month.

Read the plans page for current rates.

Who built this

A small team of engineers and lawyers built this.

We ship from Europe and work in the open.

Our founder note spells out why we started.

Where to start

How the parts fit

A browser add-on cleans text inside Chrome.

A Word plug-in handles drafts in Office.

A small desktop tool works on whole folders.

An agent protocol link feeds large models safely.

All four share one core engine and one rule set.

Words from our team

We started this work after a lunch about cookies.

One friend kept getting odd ads on her phone.

We asked why a court file leaked through a draft.

We sketched the first build on a napkin that week.

By month three we had a tiny demo for a friend.

She used it on her first case the next day.

Common questions we hear

Can the tool read scanned PDFs? Yes, with OCR.

Does it work on long files? Yes, in small chunks.

Can I roll my own rule set? Yes, save it as a preset.

Does it run offline? The desktop build runs offline.

Do you keep my files? No, the cloud build wipes after each run.

Will it learn from my work? No, we never train on inputs.

A short tour of the workflow

Upload a file or paste a snippet of prose.

Pick the entities you want gone from the draft.

Choose a method: replace, mask, hash, encrypt, or redact.

Press run and watch the side panel show each hit.

Skim the result and tweak any rule that misfired.

Save the cleaned file or send it to a teammate.