Fine-tuning a model on your own text means teaching it the personal data in that text as well. Training corpora are full of names, home addresses, email and phone numbers, national-ID numbers, bank accounts and dates of birth.
A model trained on uncleaned text can memorise that data and later reproduce a real person's details in response to a prompt that was never meant to surface them.
This guide covers a model-agnostic approach for catching personal data before training: how it is built, where it strains, and what the measurements say.
The problem is not in form fields
Personal data rarely sits in a tidy form field. It hides in customer-support transcripts, HR records, chat logs and the long free-text columns that make up the custom datasets teams fine-tune on.
It also arrives in messy, multilingual formats that no predefined schema anticipated. That is the real difficulty.
- The usual tools are bi-directional token-classification models: transformers that label each token with a personal-data type.
- Those types are fixed at training time; the schema is frozen.
- A domain-specific identifier such as an employee ID or a crypto-wallet address falls outside that schema.
- Adding one means relabelling and retraining, and the tool stays locked to one model and one deployment.
A language model reframes the problem
A language model reads its instructions at inference time. That single difference changes everything: the entities to detect, the output format and the backend all move from code into configuration.
One detector can target a new entity type by editing a prompt instead of retraining. It can run on a managed API or inside your own virtual private cloud, and it can reason about context across eight languages without a translation step.
Two design choices
Two decisions make the approach model-agnostic, and neither can be bolted on later — both have to be made at the start.
- Instruction-driven detection: all the detection logic lives in the instructions and a thin parsing layer. That keeps it independent of any single model's idiosyncrasies.
- Configurable backend: the model is reached through a uniform inference interface. Any object that takes a list of messages and returns the assistant's text satisfies it.
- The result: the same detector runs against a managed service and against an open model on your own GPU.
- Air-gapped setups: environments that cannot reach outside stay in scope for the same reason.
Customisation comes from two independent components. The first is the model, which sets accuracy, latency and cost. The second is the entity set, which defines what counts as personal data, and changing it is a one-line edit to the instructions.
The detector's four parts
The approach is built from four parts, read in the order a request flows through the system.
| Part | Its job |
|---|---|
| Prompt | Defines the schema: fifteen entity categories, one-line definitions, a do-not-flag list |
| Backend | Runs the model, through a uniform inference interface |
| Parsing | Turns the response into located spans |
| Call sequence | Ties the three together |
The job asked of the model is narrow and well defined: read the text, find every personal-data span, and label each with an entity type from the schema. The response comes back as structured JSON.
Why the model does not give positions
The detector does not ask the model for character offsets, because a language model cannot produce them reliably. The model returns only the entity type and the exact text value it found.
Offsets are recovered afterwards: each value is located in the source text with a regular expression and converted into a character range. The distinction looks small, but much of the detector's reliability comes from it.
The prompt also carries a do-not-flag list and a few optional examples. Those two additions stop the model from being over-eager and treating every proper noun as personal data; in this work a false positive can cost as much as a miss.
Collecting hallucinated labels
Language models routinely emit near-miss labels: DATE instead of DATES, EMAIL instead of CONTACT_INFO.
- Each label is re-homed onto the prompt's own vocabulary through morphology and a curated alias table.
- A label that no tier can map is not force-fitted; it is marked UNK.
- That keeps genuine hallucinations visible instead of quietly counting them as correct.
- For the honesty of the measurement, that behaviour matters more than the detection rate.
What it takes to run
The setup side is light. Python 3.11 or later plus Boto3 as the only runtime dependency is enough; the detector runs no other service.
- On the managed path you need a cloud account and model access enabled for your chosen model.
- Credentials resolve through the standard chain; a role or SSO profile is recommended over long-lived static keys.
- The detector takes a string and returns a list of located spans, ready to feed a redaction step.
- Changing the model id is a one-line job; the detector and the call site stay identical.
If you run it on your own server, shutting that host down stays your responsibility, because the detector does not manage backend infrastructure. On the managed path you pay only for the tokens you use.
How the benchmark ran
Evaluation used five public corpora, each carrying ground-truth spans, sampling roughly 10,000 rows per dataset.
In total 49,365 records and 222,114 core ground-truth spans across eight languages: German, English, Spanish, French, Hindi, Italian, Dutch and Telugu. The domains run from multilingual synthetic profiles to English HR and customer-service documents.
A predicted span is matched to ground truth only on exact overlap: start, end and label all have to agree. Precision, recall and F1 follow from there. That strict criterion lowers the scores but makes the comparison meaningful.
Aligning the labels
Comparing detectors across these corpora is harder than it looks, because the labels do not line up. Each detector and each dataset uses its own vocabulary.
For a fair comparison every raw label is mapped onto a single canonical taxonomy of twelve common entities: name, address, contact, date, age, national ID, financial, IP, URL, username, password and document number. Each detector is scored only on the scope it and the dataset both declare, so no tool is penalised for a category it never claimed.
That canonical taxonomy also defines two reporting scopes. Core F1 is the fair head-to-head number over the twelve common types. Extended-entity F1 covers dataset-specific categories such as occupation, company name or crypto-wallet address, which most off-the-shelf detectors have no notion of.
The results
The headline metric is span-level core F1. The table places managed models and self-hosted open models side by side.
- Mistral Large 3 took the top score on the managed side at 83.1%, at roughly 1.16 seconds per detection.
- OSS-GPT 20B, self-hosted, came just behind at 81.6% in a similar time.
- PrivacyFilter, the off-the-shelf tool, landed at 80.7%.
- Nova Lite 2 scored lower at 74.9% but was the fastest at 0.77 seconds.
The real message here is not a ranking. The scores sit close together, and the right choice depends on where your workload presses among accuracy, latency and cost. In a batch clean-up that runs once a day latency is irrelevant; in a live stream it matters more than a few points of score.
Seen from a regulated market
Under European or Turkish data protection rules, cleaning a dataset before handing it to a model is not only good engineering but an obligation. The most useful property of this approach is schema flexibility: local identifiers such as a national ID, a tax number or an IBAN can be defined by adding one line to the prompt.
One warning belongs here too. Turkish is not among the eight languages in the benchmark. Do not assume how accurate the same detector is on Turkish text without measuring it on your own data; name and address formats in particular are highly language-dependent.
A practical point as well: at which stage you run the detector. Scanning once right after collecting the data is not enough, because the source system keeps running and new records have to pass the same process. Treating detection as a one-off clean-up leaves every batch that arrives after the first training run unprotected.
Where to stop
Detection is not a solution on its own but the input to a next step. The spans found feed a masking or deletion step, and the real decision is made there: which data is removed, and which is replaced with a pseudonym.
The second warning is about measurement. No detector catches everything, and the best score here sits at 83%. So rather than closing the process as "cleaned", you have to put in writing whether you find the remaining risk acceptable.