How it works / Technical docs

Keys

Keys

Mechanisms only. No key material, secret values or configuration values appear in these pages, and none is needed to follow them.

The keys and what each opens

Key Made by Held by Opens or produces Destroyed
Product key-encryption key (KEK) each product's control plane that control plane only wraps every deal data key and every lookup key of that product rotated by the control plane (rewrap)
Deal data key the control plane, per deal wrapped under the KEK; unwrapped into one job's claim the deal's vault objects (server-side encryption with a customer key); the per-product salt; the identity map key; the watermark key crypto-shred: the control plane destroys the wrapped key
Deal salt HMAC-SHA256(deal data key, product label), derived in the runner per job the runner, in memory, for one job; also kept by the control plane as the narrow lookup key every keyed stand-in, keyed hash and fingerprint the package makes for the deal with the data key, unless kept as the lookup key
Narrow lookup key the control plane keeps the deal salt, wrapped under the KEK on its own label the control plane recomputes a keyed stand-in from an identifier; opens the identity map's lookup section (random stand-ins). Decrypts no vault object and doesn't open the map on its own schedule, set by the control plane's retention rules (typically the end of the buyer's contract term)
Identity map key K_map HMAC-SHA256(deal data key, product map label) the runner, per job the sealed identity map's map section with the data key
Lookup section key K_lookup HMAC-SHA256(deal salt, "identity-lookup-v1") the runner, per job; recomputable from the lookup key the map's lookup section: random stand-ins by keyed index, tokens only with the lookup key
Watermark key derived from the deal data key per buyer org and version the runner, per deliver job the per-buyer watermark on a delivery (method in the auditor edition) with the data key (attribution ends with the deal)
Report signing key an Ed25519 key from a 32-byte seed the runner deployment holds as a secret the runner the deid_report envelope signature; kid = first 16 hex of sha256 of the raw public key. No seed configured: the report goes out unsigned (alg: "none") and says so by the runner operator
Buyer recipient keys the buyer the buyer (public keys registered with the control plane) delivery files and the delivery manifest (encrypted to every registered key of the buyer org) by the buyer

The salt is the only key the package ever sees. It takes every key from its caller, refuses one shorter than 16 bytes (keys.MIN_KEY), and has no default, no module-level key and no literal key. Turning stand-ins off (set_anon(None)) produces untokenized tags ([EMAIL]), never output keyed by something guessable.

What the salt keys

Each product label is fixed per product and each runner's tests/test_salt_source.py checks that every salt it hands the package comes from HMAC(deal data key, its own label), and forbids the other product's label. The package's test_salt_labels_stay_put pins both labels: renaming one changes every stand-in that product ever delivered (or orphans every sealed map); sharing one would link the products.

Output Construction (all under the deal salt)
Keyed stand-in [KIND_<10 hex>] HMAC-SHA256(salt, "<kind>:<normalized value>"), first 10 hex
Email stand-in user_<10 hex>@org_<8 hex>.invalid (or @freemail.invalid) local part and domain keyed separately, so one company's people share an org_
Business-id token [<LABEL>_<10 hex>] HMAC-SHA256(salt, "biz:<canonical core>"), first 10 hex
Owner-identity token HMAC-SHA256(salt, "blind:<canonical term>"), first 10 hex
Git identity HMAC-SHA256(salt, "git:<identity key>"), first 10 hex
Path collision tag HMAC-SHA256(salt, …), first 8 hex
Identity graph link and grow keys HMAC-SHA256(salt, "identity-link"), HMAC-SHA256(salt, "identity-grow"); link keys are 16-byte keyed BLAKE2b, redaction fingerprints 8-byte keyed BLAKE2b
Random stand-in index HMAC-SHA256(salt, "rnd:<kind>:<normalized value>"), first 32 hex. The index is stored in the sealed map; the token it points to is CSPRNG output, never computed from the value
Date offset HMAC-SHA256(salt, "scrub date offset v1"): 1 to 52 whole weeks, sign from one bit, never zero
Realistic surrogates a sub-key HMAC-SHA256(salt, "surrogates-key:v1") drives the draws
Duplicate and fingerprint sketches keyed BLAKE2b under a sketch key from the caller

The person id in the linkage table (person_<12 hex>) is CSPRNG output in both modes.

Sealed identity map

The package hands the map over as bytes (Graph.state()) and never stores it. The reference runner seals it:

blob   {"v": 1, "alg": "A256GCM", "map": b64(nonce || ciphertext), "lookup": b64(nonce || ciphertext)}
map    Graph.state()                       under K_map     (AES-256-GCM, 12-byte random nonce)
lookup zlib JSON {index: [random stand-in, ...]}  under K_lookup
AAD    "<label>|<deal id>|<section>"       so a section can't be replayed into another deal or slot

The control plane stores the blob in parts of at most 2.5 MB. A run reads the head version, stages the new version's parts, and commits by moving the head from the version it read; a commit answers 409 when another run committed first, and the runner reads the newer map, absorbs it (nothing dropped, older person ids kept) and retries, up to 6 tries. A map in one mode refuses the other (ModeChanged). A claim whose map won't open under its key fails the job rather than starting an empty map.

Crypto-shredding

Destroying a deal's data key (the control plane deletes the wrapped key):

  • the deal's vault objects can no longer be decrypted;
  • the identity map no longer opens, so no new run can link to old ones;
  • the salt can't be re-derived from the data key, so no new keyed stand-in can be made;
  • watermark attribution for every version of the deal ends.

What survives, by design, until its own end: the narrow lookup key, if anything from the deal reached a buyer and the contract term hasn't ended. While it lives, the desk can still answer "which stand-ins could this person have become?" for a deletion request; it can't decrypt anything. Destroying it ends that too. A control plane's database backups can hold wrapped keys until their retention window ends; each product states that window in its deletion certificate.

Isolation contract

The package is shared code only. It never reaches infrastructure, keys or data of any product, and tests/test_isolation.py fails the build of every runner that vendors it if that changes:

  • No outside IO. No network, process or host modules; no environment reads; no open() outside a short per-file allowlist, each entry with its reason; every git command local.
  • No product names. No product name, domain, host, bucket, app, org, key label or claim field anywhere in the package, these pages included.
  • Caller-held keys. Every keyed API takes its key from the caller and refuses a missing, empty or short one.
  • Unlinkable output. The same input under two keys gives disjoint stand-ins, person ids, git identities, repository-file identities, owner-identity tokens and fingerprints; the same key always gives the same output.
Test What it checks
test_no_outside_io_or_environment No outside io or environment
test_allow_entries_are_used_and_justified Allow entries are used and justified
test_engine_never_imports_the_harness Engine never imports the harness
test_git_commands_are_local Git commands are local
test_product_patterns_catch_every_known_name Product patterns catch every known name
test_no_product_or_infrastructure_names No product or infrastructure names
test_every_keyed_api_requires_an_explicit_key Every keyed api requires an explicit key
test_missing_empty_or_short_key_raises Missing empty or short key raises
test_a_real_key_is_accepted A real key is accepted
test_no_module_level_or_literal_keys No module level or literal keys
test_same_person_under_two_keys_gets_disjoint_person_ids_and_map Same person under two keys gets disjoint person ids and map
test_random_stand_ins_under_two_keys_share_nothing Random stand ins under two keys share nothing
test_identity_map_round_trip_does_no_io The map comes in as an argument and goes out as a return value: nothing opened, no socket, no process.
test_salt_labels_stay_put Deal salts are HMAC(sse key, label) and map keys HMAC(data key, label), per product. Renaming a label changes every stand-in that product delivered (or orphans every sealed map); sharing one links the products.
test_different_keys_give_disjoint_stand_ins Different keys give disjoint stand ins
test_pii_covers_every_stand_in_kind_in_the_sample Pii covers every stand in kind in the sample
test_fingerprints_under_different_keys_share_nothing Fingerprints under different keys share nothing

Denied modules unless granted: aiobotocore, aiohttp, asyncio, azure, boto3, botocore, concurrent, ctypes, dbm, dotenv, ftplib, gcsfs, google, grpc, http, httpx, imaplib, mmap, multiprocessing, nntplib, paramiko, poplib, pty, requests, s3fs, smtplib, socket, socketserver, sqlite3, ssl, subprocess, telnetlib, urllib, urllib3, webbrowser, websocket, websockets, xmlrpc.

Third-party imports allowed: 7 (listed in the auditor edition).

File Grant Why
gitid.py subprocess runs git and git filter-repo on the local mirror the runner already holds; every call takes run= (custody-runner passes its sandbox) and every git subcommand is local (test_git_commands_are_local)
gitid.py open writes the filter-repo state file and reads back the stand-in list, at paths the caller passes inside its job directory
patterns.py open reads patterns.json next to this file
profiles.py open reads profiles.json next to this file
ensemble.py open reads thresholds.json (detector thresholds and merge settings) next to this file
surrogates.py open reads data/surrogates.json (name lists, reserved domains and number ranges) next to this file
tests/test_surrogates.py open reads data/NOTICE (the name lists' sources) to check it ships
sensitive.py open reads sensitive.json (rules) next to this file
bizids.py open reads bizids.json (business-id classes and formats) next to this file
workflow/score.py open reads bands.json (workflow band thresholds) next to this file
corpus/extra.py open reads coverage_cases.json and health_cases.json next to this file
corpus/redteam/__init__.py open reads the red-team cases.json next to this file (developer tool, never run by a job)
tests/test_coverage.py open reads coverage.json and the plan phase list from this package
corpus/__init__.py open reads cases.json next to this file; --snapshot rewrites it (developer tool, never run by a job)
corpus/repo.py subprocess builds synthetic repos for the git corpus cases with local git in a temp directory (test harness, never imported by engine modules)
corpus/repo.py open writes the synthetic case files into that temp directory
corpus/repo.py os.environ copied into the child git's environment so PATH resolves, with global and system git config switched off; no value is read
tests/test_isolation.py open reads this package's own source files
tests/parity.py open reads parity.json next to this file; --write rewrites it (developer tool, never run by a job)
bench/sensitive.py open reads sensitive_cases.json (labelled cases) next to this file (developer and CI tool, never run by a job)
bench/__init__.py open reads baseline.json next to this file; writes bench.json and baseline.json only into the --out directory the caller names (developer and CI tool, never run by a job)
tests/test_review.py open reads the committed cross-language vectors under tests/vectors that the control planes also check
report/bands.py open reads bands.json (deid_report band thresholds) next to this file
report/build.py open reads coverage.json and bench/baseline.json, and hashes the package's own .py and .json files for the report's engine hash
tests/test_techdocs.py open reads the technical docs under techdocs/ to check their version block matches this package
tests/test_report.py open reads report/schema.json and the committed cross-language report vectors under tests/vectors that the control planes also check

Engine docs at commit 1c5af8f39ebb. Plain-language versions: What happens to your data, the workflow score.