TL;DR: I'm building a PHI de-identification tool, and step one of doing that honestly is benchmarking it against the standard corpora: the i2b2 2006 and 2014 de-identification challenge datasets, now distributed as "n2c2" through Harvard DBMI's data portal. Step one failed. The portal's n2c2 page has said "Temporarily Unavailable" since at least 2026-07-28, registration is closed, and the old i2b2.org dataset page returns HTTP 500. Even when the door is open, access means per-user registration, a data use agreement, and an approval wait - which is why published de-id numbers are so hard to reproduce and why the field's canonical scores are 12-20 years old. I built a synthetic corpus generator instead. Here's the whole investigation, dated, plus what synthetic data honestly can and cannot replace.
Step one: you can't get the data
Benchmarking a de-identification tool sounds like the easy part of the project. The hard part is supposed to be the NLP: finding every patient name, date, medical record number, and phone number buried in messy clinical prose, so the notes can be used for research without exposing anyone. Detection is the science. Evaluation is just downloading the test set, right?
Here is how evaluation actually went for me.
Every de-identification paper of the last two decades benchmarks against the same corpora: the i2b2 de-identification challenge datasets from 2006 and 2014. If you want your numbers to mean anything to anyone, you report them on i2b2 2014. So on 2026-07-28 I went to get the data from the Harvard DBMI Data Portal, where the corpora live under their post-2018 name, n2c2 (National NLP Clinical Challenges). What I found was a notice: the n2c2 datasets are temporarily unavailable.
Fine, I thought. Maintenance happens. I built other things and came back.
As of today, 2026-08-18, nearly three weeks later, here is the exact state of the front door, checked from a fresh session:
- The portal itself is up and loads fine.
- The portal's data sets listing shows two entries. Neither is an n2c2 NLP corpus (both are 4CE COVID-related sets).
- The n2c2 NLP Research Data Sets project page - the page for the actual challenge corpora, 2006 through 2018 - carries this banner, verbatim: "Temporarily Unavailable. The n2c2 datasets are temporarily unavailable. If you are trying to access data from the 2019 Challenge, tracks 1 (Clinical Semantic Textual Similarity) and 2 (Family History Extraction) are available directly through Mayo Clinic." Below the dataset descriptions: "Registration is not open ... at this time."
- The legacy home of these corpora, i2b2.org/NLP/DataSets, returns HTTP 500.
- The companion n2c2 informational site refused my scripted requests outright with an Akamai "Access Denied" page. That one I'll hedge: it may just be bot filtering rather than downtime. But it means I cannot even verify the documentation programmatically.
No ETA. No status page. No mirror. The canonical benchmark for an entire subfield of clinical NLP is a "temporarily unavailable" banner, and "temporarily" has meant at least July 28 through August 18 so far. It may come back tomorrow; the point of this article survives either way, because the outage is only the loudest symptom of a structural problem.
What these datasets are, and why everyone needs them
Some history, because the names are confusing. i2b2 (Informatics for Integrating Biology and the Bedside) was an NIH-funded center based at Partners HealthCare that, starting in 2006, ran annual clinical NLP shared tasks on real (de-identified) hospital notes. In 2018 the challenge series was renamed n2c2, and stewardship of the datasets moved to the Department of Biomedical Informatics at Harvard Medical School, distributed via their portal. Same corpora, three names, one door.
Two of those challenges define de-identification evaluation to this day:
The 2006 challenge (Uzuner, Luo, and Szolovits, JAMIA 2007) used hospital discharge summaries in which the authentic PHI had been replaced with synthesized surrogates - including deliberately out-of-vocabulary, made-up names to punish systems that just memorized name lists. Seven teams, sixteen system runs, and the best systems scored above 98% F-measure across PHI categories.
The 2014 i2b2/UTHealth challenge (Stubbs and Uzuner, JBI 2015) raised the bar: 1,304 longitudinal records covering 296 patients, over 28,000 annotated PHI instances, annotated under a broad interpretation of HIPAA. Human annotators managed a token-level F1 of 0.927 against the gold standard; the best automated system hit a strict micro-averaged F1 of 0.936.
Notice something about both: even the "real" gold standards contain synthetic PHI. The notes are genuine clinical text, but the identifiers in them are surrogates, inserted so the data could be released at all. Keep that in mind for later - the field's ground truth has always been real prose plus fake identifiers.
That 0.936 from 2014 is, functionally, still the number. When you publish a de-id tool in 2026, reviewers ask how you compare on i2b2 2014. The corpus is twelve years old, drawn from one hospital system, and pre-dates most of what modern EHRs do to note formatting. It is also, at the moment, undownloadable.
The access process, even on a good day
Suppose the portal comes back tomorrow. What does access look like then? Per the archived instructions on i2b2.org, the datasets are "freely available" to researchers - subject to a data use agreement, and "each individual user must access the data independently" through the portal. In practice:
- Register an account on the DBMI portal.
- Sign the Rules of Conduct and the Data Use Agreement (the n2c2 DUAs are real legal documents, not click-through checkboxes).
- Wait for a human to approve you. There is no published turnaround time.
- Repeat for every individual on your team, because the DUA is per-person, not per-lab.
- Never redistribute the data - which also means never shipping it as a test fixture, never putting it in CI, never publishing your evaluation harness with the inputs included.
This is not villainy. The data is real patient prose; a DUA is the legally and ethically appropriate wrapper, and the people who built and maintain these corpora did the field an enormous service. But note the architecture: the standard benchmark for a global research area is administered by one team at one institution, through one portal, with per-user paperwork and no fallback. When that single point of distribution goes down - for maintenance, for a compliance review, for a staffing gap, for whatever is happening right now - the benchmark simply ceases to exist for anyone who doesn't already have a copy.
Why this hurts more than my project
Play the incentives forward and the damage compounds:
Published results become unreproducible in practice. A paper says "0.94 F1 on i2b2 2014." You cannot check that claim, cannot run the same test set through your own tool, cannot even eyeball the annotation decisions the score depends on. Reproducibility in de-id research is gated on a portal login, and today on a portal banner.
New tools cannot compare against prior art. The literature has twenty years of numbers on these corpora. A new open-source tool that cannot access them either skips comparison (and gets dismissed) or quotes other papers' numbers against its own results on different data (and misleads).
The canonical numbers fossilize. Because making a new shareable gold standard from real notes is brutally expensive - the 2014 corpus took double annotation, arbitration, and multiple proofreading rounds to reach that 0.927 human F1 - nobody replaces the old benchmarks. The field's reference points are frozen in 2006 and 2014 while clinical documentation, and the models reading it, changed completely.
Insiders and outsiders diverge. Groups with long-standing access or local hospital data keep publishing; independent developers and open-source maintainers evaluate on whatever they can scrape together. The people most likely to ship a de-id tool you can actually download are the least able to prove it works.
The alternatives tour
Before building anything, I did the diligence on every other door:
PhysioNet. The MIMIC family of ICU databases includes clinical notes, and PhysioNet also hosts a gold standard corpus of 2,434 de-identified nursing notes with realistic surrogate PHI (Neamatullah et al., 2008). Access requires becoming a credentialed user: identity verification, the CITI "Data or Specimens Only Research" training course, and then a separate DUA per dataset. To PhysioNet's credit, this process is documented, predictable, and actually functioning. But it is still weeks of process, still per-person, still non-redistributable. And MIMIC's notes are already de-identified with placeholders, so to evaluate a de-id tool you must first re-inject surrogate PHI - which is exactly what prior work on synthetically-identified MIMIC notes does. The "real data" path quietly becomes a synthetic-PHI path anyway.
MTSamples. MTSamples.com hosts thousands of publicly available transcribed sample medical reports across dozens of specialties. No registration, no DUA, real clinical language structure. The catch: they are samples, so they contain no PHI to find. To make a de-id benchmark out of them you inject fake PHI into the text - an approach with an established lineage in the literature. This is the honest public option, and it is the spirit my workaround follows.
Synthea. Synthea generates fully synthetic patients with medically plausible histories - fantastic for structured FHIR data. But its narrative output is template-driven fill-in-the-blank SOAP text linked to the structured record. For de-id evaluation, where the entire game is the messiness of real prose - copy-paste artifacts, headers, inconsistent formatting, abbreviations - templated narrative is the wrong distribution by design.
Newer synthetic clinical text efforts. This space is heating up: Synthetic4Health generates annotated synthetic clinical letters; ASQ-PHI proposes an adversarial synthetic benchmark specifically for de-identification; and there is active work on whether LLM-generated notes actually match real note distributions (early answer: imperfectly, and you should measure the gap rather than assume it away). None of these is yet a community-standard replacement for i2b2 2014. All of them are bets on the same thesis I ended up betting on.
The workaround: a synthetic corpus generator
So I built my own synthetic corpus generator for the PHI de-identification tool I'm building. The design is deliberately boring:
- Templates styled after real clinical note genres - discharge summaries, progress notes, radiology reports - with the section structure, boilerplate, and formatting quirks those genres actually have.
- Controlled PHI injection. Fake names, MRNs, dates, phone numbers, addresses, providers, and facilities generated and inserted at known offsets. Every injected entity is recorded with its exact span and category at generation time.
- Category balancing. Real corpora are dominated by dates and names; rare categories (fax numbers, device IDs, URLs) barely appear. A generator can oversample the rare stuff so your recall numbers on those categories are backed by hundreds of instances instead of four.
The advantages are structural, not incidental:
- Labeled by construction. No annotation budget, no inter-annotator disagreement, no 0.927 ceiling on ground-truth quality. The generator knows where the PHI is because it put it there.
- Shareable. No DUA, because there is no patient. The corpus - and more importantly the generator - can live in a public repo, run in CI, and ship as test fixtures. Anyone can regenerate the exact evaluation set from the code and a seed. That is a property the i2b2 corpora can never have.
- Scalable and steerable. Need 10,000 more notes with dates in weird formats? That's a parameter, not a grant application.
The honest limits
Here is where I'm obligated to argue against myself, because synthetic evaluation has failure modes that will flatter your tool if you let them:
Distribution shift. My templates are styled after clinical notes; they are not clinical notes. Real notes contain dictation artifacts, OCR junk, mid-sentence copy-paste, and formatting chaos that no template library fully reproduces. A recall number earned on synthetic text is an upper bound, not an estimate, of real-world recall. There is emerging work on quantifying exactly this synthetic-to-real gap for PHI taggers, and the gap is real.
Overfitting to your own generator. This is the insidious one. If the same mental model writes both the fake-name generator and the name-detection logic, the benchmark and the tool share assumptions, and you are grading your own homework. Mitigations exist - independent sources for injection values, formats the detector wasn't designed around, adversarial edge cases - but the risk never reaches zero.
It complements real-data validation; it does not replace it. My position after all this: synthetic corpora are for development, regression testing, category-level diagnostics, and public reproducibility. Before anyone trusts a de-id tool with actual patient data, it needs validation on real notes under proper governance - PhysioNet credentialing, an institutional dataset, or the n2c2 corpora if the door ever reopens. Remember, though, that even those gold standards are real prose with surrogate identifiers. The line between "real benchmark" and "synthetic benchmark" was always a spectrum, not a wall.
What better infrastructure would look like
None of this requires new science. It requires plumbing:
- A status page and a mirror. If a dataset is the reference benchmark for a field, "temporarily unavailable" with no ETA on a single portal should be impossible. PhysioNet already demonstrates the model: documented process, predictable credentialing, many datasets under one durable roof.
- Standing distribution instead of single-lab stewardship. Move canonical corpora to infrastructure whose job is distribution, with credentialing handled once per user, not per corpus per portal.
- Publish generators, not just corpora. A community-maintained synthetic benchmark - generator code plus seeds, calibrated against real data - would give the field something no DUA can: an evaluation anyone can run, extend, and verify. The recent synthetic-benchmark papers are steps in this direction; what's missing is convergence on one that leaderboards accept.
- Two-track evaluation as the norm. Report on the gated real corpus for comparability, and on an open synthetic corpus for reproducibility. Either number alone is half a result.
Takeaways
- Check the data before you plan the benchmark. The canonical de-id corpora are behind a per-user DUA on one portal, and that portal's n2c2 datasets have been "temporarily unavailable" from at least 2026-07-28 through 2026-08-18, with the legacy i2b2.org page throwing HTTP 500. Verify the current state yourself; date what you find.
- Single-institution gatekeeping of a field's benchmark is a reliability bug, independent of any outage. Per-user DUAs also mean no CI, no fixtures, no redistribution - reproducibility is structurally capped.
- The field's reference numbers are from 2006 and 2014. Best strict F1 of 0.936 on 1,304 notes from one hospital system is still the bar new tools are measured against.
- PhysioNet is the functioning real-data path - credentialing, CITI training, per-dataset DUA - but its notes need surrogate PHI re-injection for de-id evaluation anyway.
- Synthetic corpora buy you labels by construction, shareability, and category balance. They cost you distribution realism, and they tempt you into grading your own homework. Use them for development and public reproducibility; validate on real data before production.
- The gold standards were already part synthetic. Real notes, surrogate PHI. Synthetic evaluation isn't a betrayal of rigor - unexamined synthetic evaluation is.
If you work in clinical NLP: how did you get your benchmark data, and how long did access take? And if anyone has current information on when the n2c2 datasets are coming back, the comments are open.
