Measuring redaction is harder than doing it. To know whether a system caught every social security number in a document, you have to already know how many were there, and on real customer records nobody does. You can sample, and you can have a human read a subset, but you cannot check your work against an answer key that does not exist.
So we build the documents instead.
The corpus
We generate roughly 4,000 synthetic records, each one assembled from templates with fabricated values injected at known positions. Because we place every sensitive value, we know exactly where each one is.
The corpus spans the formats the product actually handles:
- Composite documents such as PDF, DOCX, PPTX, and RTF, which are not simply text. A single PDF holds a text layer, embedded images, and scanned pages, and each part has to be found and handled on its own terms.
- Structured and plain text such as CSV, XLSX, JSON, XML, HTML, and TXT, where a value’s meaning often comes from the column it sits in rather than the words around it
- Images in PNG, JPEG, and TIFF, where a record is a photograph or a scan and has to be read before anything can be redacted
- Audio in WAV and MP3, where names are spoken aloud and card numbers are read digit by digit
The values are fake. The shapes are not: formats, field positions, and the ways identifiers appear mid-sentence are drawn from what these documents actually look like in each domain.
What we can measure with it
With ground truth, the numbers stop being estimates:
- Recall, meaning how many of the identifiers we planted were found. A miss here is a leak.
- Precision, meaning how many of the redacted spans were actually sensitive. Failures here are over-redaction, which quietly destroys the document’s usefulness.
- Boundary accuracy, meaning whether the redaction covers the whole value and nothing more. Catching eleven digits of a nine-digit number spanning a line break is not a pass.
Splitting recall by entity type is where the work actually gets directed. An aggregate score hides that names in running prose behave nothing like account numbers in a table.
The hard cases we plant on purpose
Generating documents lets us construct the situations that are rare in any one real corpus but certain to appear across all of them:
- A name that is also a common word, such as a patient named April or a counterparty named Case
- The same person written six ways across one file: full name, initials, surname only, misspelled, in a signature block, in an email address
- Identifiers split across a line break or a page boundary
- Numbers that match a format but are not that thing, such as an order number shaped like a social security number
- Scanned pages where OCR reads the value imperfectly, and audio where a name is transcribed phonetically
Each of these gets its own slice in the corpus, so a regression in one shows up as a specific number rather than a rounding error in the total.
How the answer key handles a mixed file
A benchmark that scores whole files hides where a failure actually happened. The same PDF can carry a text layer that redacts cleanly and an embedded scan of the same page that does not, and one number for the file cannot tell those apart.
So the answer key records modality per part rather than per document. A PDF decomposes into its text, its embedded images, and its scanned pages, and each part is scored on its own.
That separation is what makes a failure diagnosable. When an identifier survives in a scanned page but is caught in the surrounding text, detection is working and the reading step is not. When a spoken name is missed in audio but found in its transcript, the problem is transcription rather than recognition. Generating the same records across formats gives us that comparison deliberately instead of by luck.
When we run it
We run the corpus when the detection pipeline changes. A drop in recall for a single entity type is the signal worth acting on, because it names the failure precisely enough to chase: not “accuracy fell” but “account numbers in table cells stopped being found.”
It is also how we check the deterministic-output property. The same document through the same policy should produce the same redaction, and re-running a fixed corpus after a model or ruleset change is what shows whether that still holds.
Synthetic documents do not replace evaluation on real ones. What they give you is a floor: a fixed, repeatable measurement that catches a regression before it reaches a document that matters.
The generator and the scoring harness are open source at nvisycom/synthetic. Since every value is fabricated, the corpora carry no real data, and you can generate your own and score a redaction system against them.