My fine-tuned model scored 100%... The benchmark was lying

I fine-tuned Mistral 7B on my laptop to detect personal data in log lines and support messages. On my first test set it scored 100%. Perfect. Every single line classified correctly.

I did not publish that number, because the same test set gave few-shot prompting 94%, and a six-point gap over a prompt you can write in five minutes is not a reason to fine-tune anything. The honest conclusion looked like "this was a waste of an afternoon."

Then I threw my test set away and rebuilt it from real public data. The fine-tune dropped to 95%. Prompting collapsed to 66%.

Same model, same code, same training recipe. A 6 point gap became a 29 point gap, and the conclusion flipped completely. My benchmark had been choosing my answer for me, and it had chosen wrong.

This article is the whole run: what LoRA actually does, how to build a dataset that does not lie to you, the exact commands, and the measured results. Everything is reproducible from the repository on any Apple Silicon Mac.

Is LoRA real fine-tuning?

Worth settling first, because "LoRA is not really fine-tuning" comes up constantly.

It is. LoRA trains the model with gradient descent on your data exactly like full fine-tuning. The difference is which weights move. Instead of updating all 7 billion parameters, it freezes them and learns two small low-rank matrices per targeted layer. Their product approximates the weight update that full fine-tuning would have made, and it can be merged back into the base weights afterwards, giving you a genuinely different model.

In my run, that meant 0.145% of the parameters were trainable: 10.5 million out of 7.25 billion. That is the entire reason this fits on a laptop, and why the output is a 42 MB adapter file instead of a new 4 GB model.

What it is not is prompting or retrieval. The weights actually change. The honest caveat is that full fine-tuning can push a bit further on hard tasks, at ten to a hundred times the memory cost. For teaching a model a format, a taxonomy, or a behaviour, LoRA is what practitioners actually ship, and it is what the fine-tuning APIs from the major providers run under the hood.

The task and the hardware

The model gets one line of text, in English or French, and must answer with strict JSON:

{"pii": true, "types": ["email", "name"]}

Six types: email, phone, name, iban, address, dob. Empty list when there is nothing.

This is a real problem, not a toy. Personal data leaks into logs, staging dumps and exports far beyond your users table, which I wrote about in PII and data masking, and knowing where it is is the precondition for retention and deletion that actually works.

Hardware: a MacBook with an Apple M5 and 16 GB of unified memory. Base model mlx-community/Mistral-7B-Instruct-v0.3-4bit, running on MLX, Apple's array framework. No cloud, no API keys, total cost 0 EUR.

The part nobody writes about: the dataset

Every tutorial shows you mlx_lm.lora --train. That command is four minutes of work. The dataset is the other three hours, and it is where the result is actually decided.

Attempt one, and why it lied

I generated 800 examples from my own templates: log lines with fake emails, support messages with fake IBANs, plus hard negatives full of UUIDs and invoice numbers so the model could not just flag anything that looks like an identifier.

It gave the fine-tune a perfect score. The reason is obvious in hindsight: the test set came from the same templates as the training set. The model had seen every sentence shape before. I was measuring memorisation of my own imagination.

If your fine-tune scores 100%, your test set is too easy. That is not a nice problem to have, it is a broken measurement.

Attempt two: real data, and a licensing trap

So I went looking for public corpora, and immediately walked into two problems that are worth more than the rest of this article.

The obvious choice is ai4privacy/pii-masking-200k, which everybody cites. Its licence is dual: free for individuals, non-profits, and companies with three staff or fewer, paid otherwise. A blog that markets a consulting practice is commercial use. Depending on your situation you may be fine, but "everybody uses it" is not a licence.

For negatives I grabbed LogHub, 32,000 lines of real production logs from 16 systems. Two problems. Its licence covers "research or academic work" only. And, far worse for an article about detecting personal data, several of those systems contain real personal data:

• BGL exposes real researcher account names in /home/ paths from national laboratories
• Mac logs contain a real email address and a real home directory
• Linux and OpenSSH carry real usernames from live attack traffic

My screening regex caught four email addresses and missed all of it, because a username inside a file path does not look like contact data. I had built a training set that teaches a model that real people's names are not personal data. That is worse than no model at all.

Both sources were thrown out. The final pipeline uses two Apache-2.0 corpora: kiji for positives, which covers all six types including IBAN and date of birth in English and French and ships its own train/test split, and witfoo syslog for negatives, 155,000 lines of real firewall and system logs.

The shortcut that would have faked the score again

Here is the trap that would have quietly ruined everything. Positives are business prose. Negatives are raw syslog. A model can separate those two by writing style and score brilliantly without ever learning what personal data is.

So every quadrant has to exist:

The prose negatives are built by rewriting each annotated span into a generic role word, so "contact Alice Dupont at alice@example.fr" becomes "contact the customer at the support address". Same sentence structure, same vocabulary, same language, no personal data.

Then I found the leak inside my own fix. Filler phrases like "the customer" appeared only in negatives, so they became a perfect giveaway. The model could learn my filler vocabulary instead of the task. So 10% of the rows are hybrids: role words everywhere, except one real value left in. Now the filler carries both labels and is useless as a signal.

Four defects in the public data

Public datasets are not clean. Screening mine turned up:

Blanket filtering costs three quarters of the corpus. Most kiji rows mention a passport or an SSN somewhere, types I do not model. Dropping those rows left 4,180 usable. Replacing just those spans with role words instead left 15,810.

The annotations are incomplete. Kiji has a coreferences field, and it is empty on every single row. So a later mention survives: "Alice Dubois" is annotated, but "Dubois a signé" three sentences down is not, and that real surname lands inside a "no personal data" example. Fix: screen every negative against a 1,519-token name vocabulary harvested from the corpus itself.

4.7% of the French rows are encoding-corrupted. Accented characters arrive as NUL bytes, so étude is stored as \x00tude. Train on those and you are teaching the model mojibake French. 406 rows dropped.

4% of the syslog carries account names in prose, like Accepted password for johndoe. Too few to be worth parsing, so they are dropped wholesale rather than mislabelled as clean.

Final dataset: 8,000 train, 448 validation, 800 test. 4,572 positives against 4,676 negatives. 2,544 French rows. Zero overlap between splits, verified.

The Mistral gotcha that costs you an afternoon

My first smoke test crashed:

jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/...

Mistral's chat template rejects a standalone system role. Unlike Llama or Qwen, it wants strictly alternating user and assistant turns. So the instructions have to ride on the first user turn:

{
  "messages": [
    {"role": "user", "content": f"{SYSTEM_PROMPT}\n\nLine: {text}"},
    {"role": "assistant", "content": '{"pii":true,"types":["email"]}'}
  ]
}

The dangerous part is not the crash. It is that if you fix this in your evaluation script and forget your dataset builder, training and inference use different prompt shapes, and your adapter looks broken for no visible reason. Build both from the same function.

Training

python -m mlx_lm lora \
  --model mlx-community/Mistral-7B-Instruct-v0.3-4bit \
  --train --data ./data \
  --fine-tune-type lora \
  --batch-size 4 --num-layers 16 --iters 500 \
  --learning-rate 1e-5 --max-seq-length 512 \
  --mask-prompt --grad-checkpoint \
  --steps-per-report 50 --steps-per-eval 250 --save-every 500 \
  --val-batches 25 --seed 42 \
  --adapter-path ./adapters 2>&1 | tee training.log

Why these values:

--mask-prompt is the one you must not skip. It computes the loss on the answer only, not on the input line. Without it, most of the tokens the model is learning to predict are the log line itself, which is not the task.

--batch-size 4 --num-layers 16 came from measurement, not guesswork. A first run at batch 1 with 8 layers peaked at 4.8 GB on a 16 GB machine, so there was room to double both.

--learning-rate 1e-5 is the consensus for small datasets. 1e-4 oscillates, 1e-6 barely moves.

--grad-checkpoint trades compute for memory, and --save-every matters because you want the best checkpoint, not the last one.

The loss curve is the interesting part:

Almost everything happens in the first 250 iterations. The next 250 bought a 3% improvement, so I stopped there rather than running the 2,000 I had planned. If you take one operational lesson: watch validation loss and stop when it flattens, because "train longer" is mostly a way to spend electricity.

Final cost: 45 minutes, 6.0 GB peak memory, a 42 MB adapter, 0 EUR.

Results

Every mode uses the same prompts, the same test set and temperature 0. "Few-shot" means six worked examples in the prompt, which is what a sensible engineer tries before reaching for training.

400 held-out rows

Look at few-shot's precision: adding six examples made it worse than zero-shot, 104 false positives against 75. It found more personal data and cried wolf far more often.

The per-type breakdown shows why:

IBAN is the story. Six examples cannot teach a model the boundary between an IBAN, an invoice reference and a whsec_ webhook secret across two languages. Few-shot scores 0.194 there, worse than saying nothing. The fine-tune reaches 0.950, because 4,000 examples of that boundary is what it takes.

30 hand-written lines the model has never seen

I also wrote 30 lines by hand, in phrasings the corpora never produced, as a final honesty check:

The fine-tune got all 30. Small sample, so I would not put "100%" on a slide, but it did not fall apart off-distribution, which was the real question.

It is also cheaper to run

The fine-tune is 1.8x faster per line than few-shot, 0.91 seconds against 1.67. The six examples are gone from the prompt, so every single inference is shorter, forever. Better and cheaper is a rare combination.

When you should not do this

Fine-tuning teaches behaviour, format and taxonomy. It does not teach facts.

If your problem is "the model does not know our internal documentation", fine-tuning is the wrong tool and retrieval is the right one. Facts change; weights do not. You will retrain forever and still get confident wrong answers.

And prompting deserves a fair trial first. On my synthetic dataset it genuinely tied the fine-tune. It only collapsed when the task got hard enough that six examples could not express the rules. That threshold is the actual decision point, and you cannot find it by reading blog posts, including this one. You find it by measuring both, which costs an afternoon.

The lesson that generalises

The technical work here was easy. MLX is excellent, the command is one line, and it ran on a laptop while I did something else.

The hard part, and the part that decided the outcome, was the data: the licence that quietly excludes commercial use, the "clean" log corpus full of real usernames, the empty coreference field, the NUL-corrupted French, and the style shortcut that would have handed me a beautiful meaningless number.

I got two completely different answers to the same question, on the same model, on the same day. The only thing that changed was the quality of what I measured against. Before you trust any fine-tuning result, including the ones in this article, ask what the test set is made of.

Reproduce it

Everything is public, including the training log and the raw predictions:

git clone https://github.com/jguillaumesio/lora-pii-detection-mlx
cd lora-pii-detection-mlx
python3 -m venv .venv && .venv/bin/pip install mlx-lm datasets
.venv/bin/python build_dataset.py

The dataset is not committed. It rebuilds from the two Apache-2.0 corpora with that one command, so nothing is redistributed that should not be. The 30 hand-written test lines, the results and the training log are all in the repo.

Originally published on jguillaumesio.com. I write about payments, AI agents in production, and running SaaS infrastructure without a platform team.