Detectors

Detection has four layers, applied to every string as written and to its decoded view:

1. Rules (detector A). patterns.json: credential formats, labeled secrets, personal-data patterns with validators, header-driven column rules, name rules, plus the company rule (capitalized words before an unambiguous legal suffix). Always on; no model.
2. Ensemble (detectors B and C, language model A). Pinned models the reference runner loads and hands to ensemble.py as plain callables. Off when no runner loads them; then name model A is the name detector.
3. Owner identity (blind.py): the deal's own terms (company, brands, domains, people) in every variant, when the control plane sends them.
4. After the scrub: the inline residual gate (patterns.residual, the same rules over the bytes about to ship), the independent re-check (separate code, below), and the release gate's human review (measurement.md).

Rules

patterns.json 1.4.0.

Validators are checksums and structural tests (checks.py: Luhn for cards, IMEI and Canadian SIN; ISO 7064 mod 97 for IBANs; ABA routing; Verhoeff for Aadhaar; TFN weights; German and Dutch VAT; NPI and VIN check digits; IPv6, MAC, UUID, coordinate and plate shape) plus a few in patterns.py (personal email, URL parameters that name a person, international phone, has-a-digit for labeled ids). A pattern hit whose validator fails isn't redacted.

Ensemble logic

ensemble.merge, for every text a detector covers (the language model routes each block; a detector with a language list runs only on those languages; name model A reads English only):

• Union. A hit at or over its kind's threshold for that detector is redacted, whichever detector found it. Kinds a detector reports without a threshold in thresholds.json are ignored.
• Merge. Overlapping kept hits become one span covering all of them. Its kind is the one first in precedence (a checksummed card beats a name), its score the best score for that kind, by the detectors in it.
• Disputed. A kept span that another detector which ran and covers that kind didn't find. Its stand-in goes into the watch set, and the output sampler puts windows showing it first in the desk's review queue.
• Near. A hit at or over its detector's review floor but under its threshold, overlapping no kept span: not redacted, sent to desk review with its context.
• Only adds. Rule hits are redacted on their own path; the ensemble never removes one.
• Blocks. Short texts are joined into blocks and long ones cut into overlapping windows, so one model call covers many lines. Results are cached per string.
• Columns. For a column whose header says nothing, a sample of its values goes through the detectors; a kind that wins enough of the sample claims the column.
• Failure. A detector exception is counted under its alias (never with its message). Under a profile with no fallback (strict) it raises, and the runner holds the file (detector_failed). Otherwise the other detectors' results stand and the report says which detector failed.
• Third-party companies. Kind org. The profile decides: tokenize (strict, standard) or keep (b2b). The seller's own names are never third-party; on owner-identity deals they are the owner pass's job.

thresholds.json 1.0.0.

• Precedence when overlapping hits merge (first wins the kind): secret, card, iban, ssn, uk_nino, ca_sin, au_tfn, in_aadhaar, in_pan, gov_id, mrn, routing, account, email, phone, ip, dob, address, username, name, org
• Kinds the package's own rules (detector A) report: secret, card, iban, ssn, uk_nino, ca_sin, au_tfn, in_aadhaar, in_pan, gov_id, mrn, routing, account, email, phone, ip, dob, address, username
• Default countries for country-filtered recognizers: us, uk, ca, au, in
• Languages routed to the multilingual detectors: en, de, fr, es, it, pt, nl
• Block size: 1500 characters per model call, 120 characters of overlap
• Column inference: 12 sampled values per column, at least 3, a kind wins the column at a 0.6 share; kinds name, account, org, address, phone, email, gov_id, iban, card
• Watch set cap 20,000 stand-ins; near-miss review items kept per run: 40; context shown: 60 characters

Models

Reports and these pages name models by alias. The tree hash is the one every report carries per alias in engine.models and models[]: the sha256 the reference runner computes over the installed files (for packages) or the weights directory (for weights), checked again before any data moves.

Lock-file commitments (sha256 of the file bytes at this version; the auditor edition prints the files):

What each one is, in plain terms:

• Detector A: the package's own rules and company rule. No model.
• Detector B: a recognizer engine of pattern and checksum recognizers with context words, filtered by the deal's countries, plus phone-number parsing. It runs on a slim tokenizer pipeline with its own entity recognition off: no remote, cloud or large-language-model recognizers.
• Detector C: a multilingual span model (about 205M parameters, CPU, full precision; an int8 quantization was measured to change 46% of outputs and rejected). Names, organizations, addresses, account and government ids.
• Name model A: a small English named-entity model, the name detector when no ensemble runs.
• Language model A: language identification in a low-accuracy trigram mode, to route blocks to detectors.
• Verifier A (optional): a token classifier from a different model family than detector C, used only by the re-check for review counts; never baked into the image, never a hold, loaded only when every pinned file matches its hash.
• Verifier B (reserved): a candidate backup verifier from the same family as detector C (so weaker as an independent check), under a custom license that needs review before adoption. Not used.
• Speech model (audio and video): multilingual speech recognition, int8, CPU. Its transcript feeds the same detectors; detected spans are toned out of the audio.
• Face detector (images and video): a small face detection model, CPU. Faces are blurred under every profile whose faces is blur.

A baked model that doesn't match its hash fails the job for name model A; for an ensemble member it fails a strict job and drops the member otherwise (reported). A missing model fails a strict job; otherwise the fallback path runs and the report says so.

Full manifest

The full manifest (model names, versions, sources, per-file hashes, runtime packages) is in the auditor edition only.

Benchmarks

python -m jobs.bench_adapter --out DIR --check (CI's bench job, with the pinned weights fetched) runs the synthetic benchmark corpus through the reference runner's real scrub path in every modality and compares it with bench/baseline.json. A gated metric below the baseline minus the tolerance, or a file leak rate above it plus the tolerance, fails the job. Text, tables and JSON are always gated; mail, office files, born-digital PDFs and healthcare files are gated when their tools are present; image and scan numbers depend on the OCR build, so they are reported and not gated.

Metrics, per entity kind:

• Recall (overlap): share of planted spans (text, tables, JSON) that any detection touches. Recall (exact): that a detection matches at both boundaries. Keyed by the planted kind.
• Precision (overlap): share of detections that touch a planted span, keyed by the detector's kind. Over-redaction lowers it.
• Removed: share of planted values that don't survive in anything a buyer can pull out of the output (OOXML parts and attribute values, PDF strings, decoded bytes, text the runner reads out, and for text drawn in an image, whether its box was blacked out). A held file leaks nothing.
• Leaked / Held: values that survived, and values in files that were held.

bench/baseline.json v1: corpus ca9a55297689575d, seed 1, regression tolerance 0.02. Recorded at package 0.6.0, patterns 1.3.0. The baseline predates the current rules (patterns 1.4.0, package 0.6.0). CI's bench job fails on any gated metric below these numbers minus the tolerance, so they are floors, not this version's measurements.

Weighted by planted values across kinds. Per-kind results are in the auditor edition.

Per modality and file type (a file leaks when any planted value survives in it):

Modalities in each mode's baseline:

• ensemble: audio, doc, face, image, json, mail, pdf, qr, scan, table, text
• ner: audio, doc, face, health, image, json, mail, pdf, qr, scan, table, text
• regex: json, table, text

Per-modality and per-file-type results are in the auditor edition.

Known caveats: the corpus is synthetic and generated by the package (bench/gen.py); real data is messier. Notes on individual baseline rows are in the auditor edition.

Sensitive attributes, per category, on labelled cases plus a held-out set written after the last tuning pass and never tuned on (read the held-out recall as the expected rate on new phrasing):

ner (cases 00d9dbd1af29a75b)

regex (cases 00d9dbd1af29a75b)

Per-category results are in the auditor edition.

Independent re-check

recheck/ is a second detection pass over every byte about to ship, written apart from patterns.py so a bug in the scrub's path isn't repeated: its own formats, its own checksums (not checks.py), its own decoder (not textnorm.py) and its own stand-in recognizer, which blanks what the scrub left before any rule runs. It rewrites nothing. A hit holds the file (recheck_hit); an unreadable shipped file is held (recheck_unreadable). With verifier A loaded, the re-check also hands it a bounded slice of text it already passed; verifier hits at or over the floor count as review items, never holds.

Re-check version 1; kinds it reports: secret, email, phone, ssn, health_id, card, iban, ip, dob, date, url_param.