Architecture and flow

The path data takes

intake ──► custody ──► runner: pull ──► runner: scrub ──► gate ──► release ──► runner: deliver ──► buyer
 owner      control     raw/ in the      raw/ → clean/     inline    control      watermark, encrypt
 connects   plane,      deal's vault     one source per    per file  plane:       to the buyer's key,
 or         deal key    prefix           job               + re-     output       Merkle root
 uploads    wrapped                                        check     sample
            under KEK                                              review

1. Intake. The owner connects a source (database, bucket, SaaS account, mailbox, repository) with a credential they create in their own account, or uploads files. The control plane seals the credential and creates the deal's data key, wrapped under that product's key-encryption key (KEK). Neither the package nor the runner ever sees the KEK.
2. Custody. One control plane per product owns the deal, its vault prefix, its keys and every job. It hands a runner one job at a time through a versioned claim and receives counts, hashes and reports back in the finish call. The claim and finish JSON is the only interface between runner and control plane.
3. Runner. One machine runs one job and is destroyed. Job types in the reference runner: quantify (sizes and counts, no content leaves), pull (source to raw/), scrub (raw/ to clean/), manifest, deliver (and a buyer-pull variant), reconcile and trace. Boot order: lock outbound traffic to the control plane only, claim, re-lock to the hosts this job type needs, pin them, drop DNS and IPv6, drop to an unprivileged user with no-new-privileges, then work. Details in threat-model.md.
4. Scrub. The engine (this package) does detection and rewriting; the runner does IO, parsing of hostile formats in a sandbox, media, models and the report. One scrub job covers one source.
5. Gate. Every byte the scrub writes passes the inline residual gate before it is uploaded, then the independent re-check, then (when the deal names owner terms) the owner-identity check. A hit holds that file; nothing is ever "fixed and shipped" without passing again. Held files are listed by path hash and reason.
6. Release. The control plane's release gate: the desk reviews a random sample of the delivered output and the version releases only when the profile's thresholds hold (measurement.md).
7. Delivery. A deliver job watermarks per buyer and version, encrypts to the buyer's recipient key, and records a Merkle root over every delivered file. The signed deid_report travels with it.

Engine and runner

A runner that vendors the package gets the engine column on every sync. The runner column is each runner's own code: which of those features another runner wires, and with which switches, is that product's documentation. Everything these pages call "the reference runner" is custody-runner at the same commit as this package.

Scrub order (reference runner)

jobs/scrub.py run():

1. Verify every baked model against its pinned hash (jobs/models.py); under a profile with no fallback, a mismatch or a missing baked model fails the job.
2. Activate the profile, load the ensemble members, run the canary self-test on every scrub path (measurement.md). Under random stand-ins the canaries run on a throwaway graph.
3. Open the deal's identity map when the claim carries one, then set the k-anonymity tracker, the stand-in salt, the date rule and the business-id map.
4. Per file, in order: the owner's never-share path rules (before download) → line formats stream from the vault through the scrub and the gate writer with no disk; everything else downloads to the work directory under a hashed name → second credential scanner (counts providers so the owner knows what to rotate; inside the sandbox it has no network, so nothing is verified against a provider) → malware scan → macro check → kind → healthcare format check → the kind's path (below) → upload through the gate.
5. After the last file: normalized tables (when the claim asks), the workflow score (when the claim asks), the linkage table and sealed map, the scrub report, the deid_report, and the manifests (scrub-report.json, forensics.json, held.jsonl.gz).

Modalities

Paths that ship (folders, file stems, archive members) are scrubbed like text under every profile (paths.py), extensions kept. File times follow the profile's date rule.

Unpacking and normalization

• Decoded view (textnorm.view). Every detector also reads a normalized view of each string: NFKC, zero-width and bidi controls dropped, combining marks dropped, homoglyphs in mixed-script words mapped, percent-encoding, HTML entities, quoted-printable and bracketed at/dot undone, base64 runs that decode to text decoded. Each replacement maps back to the original span, and the stand-in is keyed on the decoded value, so an encoded email and its plain form get one stand-in. The evasion count says how many values only the view found.
• Containers. Archives, mailboxes, git bundles, Office packages and PDFs are opened and every inner part goes through its own kind's path. Nesting is bounded (archives 2 levels, Office packages and decoded views by their own depth limits below).
• Normalized tables (claim switch, off by default). At the end of a scrub, CSV and TSV become typed JSONL tables and JSON arrays of objects become JSONL tables, each with a stats-only schema (column, inferred type, null rate, stand-in rate, distinct-count estimate; never values). They go through the same gate; a table that trips it is held (normalize_residual).

Limits

• Images: at most 512 MiB and 150,000,000 pixels, 30 frames; scanned PDFs up to 80 pages; 3 OCR redaction rounds with 3 px padding.
• Video: frames sampled at 1.0 fps plus scene changes over 0.3, at most 3600 frames at 1920 px wide; face boxes padded 0.15; bleep padding steps (0.15, 0.4, 0.8) s; at most 2000 bleeped ranges and 1500 boxes per file.
• Decoded view: up to 3 nested decodings and 256 decoder calls per string.
• Re-check: 1 MiB chunks with 512 characters of overlap; a verifier hit counts for review at score 0.9 or more.
• Healthcare PHI signal: threshold 25; weights (points per hit, cap) files 25/4, mrn 3/10, npi 3/10, icd 2/15, cpt 2/15, terms 1/20; Part 2 terms counted apart, part2Over at 3 or more.

The reference runner's sandbox adds per-tool CPU, file-size and output limits and a wall clock; the healthcare worker runs each file under 600 CPU seconds, 3 GB of memory, 2 GB of output and 900 seconds of wall time. A file over any limit is held with a limit_* reason; the job carries on.