Measurement and the release gate

Five independent measurements, from cheapest to strongest:

1. Benchmark on the synthetic corpus, per engine version (detectors.md). Says how the detectors do on data like the corpus, not on a given deal.
2. Canaries in every scrub job, before any data moves.
3. Inline residual gate and re-check on every file about to ship.
4. Red teams: an owner-identity set (reference runner) and an evasion set (package).
5. The release gate: a desk review of a uniform sample of the delivered output, with an exact miss bound.

Corpus

Owner-identity red team (reference runner, tests/redteam/): 28 planted identifiers across 6 source types, 6 deal terms; every one must be caught.

gap cases record known misses as such (the case is asserted to still miss, so fixing one fails the test until the case moves to the passing set). miss cases in the red-team set say what an evasion does that no detector handles. CI keeps every corpus case passing except the gaps.

Canaries

jobs/canaries.py, before the job reads any real data. Fresh fake values (new each run) are planted in a shadow file per modality (text, CSV, NDJSON, mail, docx, xlsx, PDF, an image with text, a video with on-screen text and subtitles, a spoken call, git history) and run through the real scrub path: dispatch, the document rewrite, OCR boxing, the mail and git rewrites, the A/V pass, the gate and the re-check. Uploads go to memory only.

• Fails the job: any canary value in what the shadow would upload, a shadow file the re-check holds, or a re-check that can't see the raw canaries in a modality it reads.
• Held: a model or tool that path needs is missing, so nothing of it would ship.
• Skipped: the modality's tool isn't installed, or the media switch never runs that path.

Names are reported (ok or missed), not failed on. The report carries the exact 95% upper bound on the canary miss rate over every value checked. It is a bound on synthetic values, not on the deal's data.

Residual gate and re-check

The residual gate scans the bytes the scrub wrote with the rules and holds the file on a hit (residual; the report's rescanClean is true only when no file hit). The re-check (jobs/recheck.py, recheck.py in the package) reads every file again with its own readers per modality, so a reader bug in the scrub isn't repeated: bytes as written for text and records; its own unzip for Office (text per paragraph, cell and row, plus every attribute value); text extraction in the sandbox for PDF; OCR again with a different segmentation for images; the rewritten log and patches for git. A hit holds the file (recheck_hit); a document it can't read back holds it (recheck_unreadable).

Re-check version 1; kinds it reports: secret, email, phone, ssn, health_id, card, iban, ip, dob, date, url_param.

Ensemble members and verifier A also read a bounded slice of each text file the re-check passed. What they flag is a review count by kind, never a hold. Verifier A is optional and absent from the published runner image.

Red teams

• Owner identity (reference runner, tests/test_redteam.py): for confidential-seller deals, the seller's identity planted the ways real data carries it (company names and variants, domains, signatures, text in images, identifying paths, database, repository and channel names, typos, initials). Every planted identifier must be removed from everything that leaves the scrub or raised to the desk as an identity-review item, which blocks release. CI fails below 100%.
• Evasion (package, corpus/redteam/cases.json): spaced-out letters, nested encodings, homoglyphs and similar. Runs on demand (python -m datacool_scrub.corpus --redteam); CI only checks the cases are well-formed and distinct from the tuned corpus. Most cases are expected misses with the reason recorded.

Both sets were written by the same author as the rules. An outside red team has not run. Not live.

Release gate

The runner takes a uniform random sample of the delivered clean text (review.OutputSampler): every delivered file is cut into fixed windows and one reservoir runs over all of them, so a file is sampled in proportion to its size. A held or dropped file's windows are taken back out. A second reservoir collects windows showing a stand-in the ensemble's detectors disagreed on, for the desk to look at first; it doesn't touch the uniform sample. Snippets travel only in the job result and are sealed with the deal key by the control plane. The desk marks each uniform-sample snippet clean or a miss, with a severity.

Output sample: 240-character windows, 300 per scrub job by default (the claim can ask for up to 2000).

One-sided exact (Clopper-Pearson) upper bounds at 95% confidence, from review.miss_bound.

miss_bound(k, n) is the exact one-sided Clopper-Pearson upper bound; for k = 0 it is (1 - \alpha^{1/n}), which the rule of three ((3/n)) approximates. When the desk reviewed every window (a census, flagged census in the report's missBound), the control plane reports the exact share instead of a bound. The control planes carry a TypeScript copy checked against tests/vectors/miss_bound.json.

The thresholds the gate applies are control-plane configuration, not engine data, so they aren't rendered here. On one control plane's development branch (gate config version 1, 95% confidence) they are: under strict, at least 300 reviewed windows with no high-severity miss and an upper bound of at most 1%; under standard and b2b, at most 2%; under HIPAA profiles, no miss of any kind. The gate is behind a control-plane switch. Not live in production at the time of writing; check the control plane's own docs for the current state.

What the bound means: with n windows reviewed and k misses, the true share of 240-character windows of delivered text containing a miss is below the bound with 95% confidence, provided the sample is uniform and the desk's marks are right. It doesn't bound misses in what the sample can't show (held files, images without text, re-identification by linkage).

Independent re-check by a different model

Verifier A is a token classifier from a different model family from the ensemble's span detector, pinned by hash like the rest. It is optional: the reference runner's image doesn't bake it, the job never requires it, and it never holds a file. Its results appear under recheck.verifier when it ran. Not live in the published image.

Report quality bands

report/bands.json 1.0.0. Metrics from the desk's output sample: missUpper95, highMisses, precision.

Privacy

Quality

Provenance

Each area's bands are tried top to bottom; a band applies when every condition it lists holds. An unmeasured metric (null) fails its condition, so the band shown is the best one the measurements already prove. When an unmeasured metric is all that stands between a report and a higher band, the band is marked provisional and names that metric. A metric listed as not applicable passes.

Not measured at this version: precision (desk-marked over-redaction in the sample; always null). The miss bound is null until the release gate runs.

Motivated-intruder test

Reported in privacy.intruder (schema 1.1.0), not banded.

• Desk part: the control plane passes the desk's result in the claim (n, identified, unsure, upper, confidence; the bound counts unsure as identified). The desk tries to identify a person or the business in a sample of the delivered text. Until a desk result exists the basis is "not yet measured" and the report's limitations say so. Whether a control plane runs it today isn't something this code shows.
• Surrogate distinguishability (intruder.py), only when the deal uses realistic stand-ins: can a real name be told from the surrogate that replaced it? A character 1- to 3-gram naive Bayes trained in-process on the run's own (real name, surrogate) pairs and scored on held-out folds (auc), and a lookup against the published surrogate name lists (listAuc). 0.5 is chance. Needs at least 30 pairs, else "not yet measured"; "not applicable" with tag stand-ins. Names stay in memory; only the scalars leave.
• Realistic stand-in emails and phone numbers use reserved domains and number ranges, so they are recognizable as stand-ins by design; the report says so when they're used.