Engineering deep dive: tamper-evident evidence at scale

What audit-ready means once the ledger is large¶

Every governed action in KLA lands in a per-tenant, append-only immudb ledger: audit entries, policy decisions, gate evaluations, policy-pack snapshots, and signed governance receipts. On a busy tenant that ledger grows without bound, and an export request arrives with a scope: one execution, one decision, one policy, or a full audit window.

At small volume, an export is a query plus an archive. At large volume, three quiet failure modes appear. A server-side scan cap truncates a result set without an error. A long-running collection sees writes race past its cursor. A partial export looks identical to a complete one, because absence leaves no artifact.

The design answer is to treat completeness as a first-class, testable property of collection, and integrity as a first-class, testable property of the sealed output. The two are enforced by different mechanisms and verified by different checks, and the rest of this post keeps them separate.

The bundle on disk¶

A Sealed Evidence Bundle is a zip archive with a fixed layout. Everything a verifier needs travels inside it: the manifest, the detached signature, the public keys, both anchor proofs, and the artifacts themselves.

The manifest records the export identity and scope, the requesting principal, policy snapshots with their pack hashes, the full artifact list with per-file SHA-256 digests and sizes, the Merkle root over that list, the redaction profile, declared omissions, and an integrity block naming the hash and signature algorithms and the anchor set.

<bundle-dir>/
  bundle.json          seal summary: digests, signer set, anchor references
  manifest.json        full manifest: population, artifacts, integrity block
  manifest.jws.json    detached ES256 JWS over the manifest digest
  keys/jwks.json       public verification keys embedded for offline use
  timestamp.ots        OpenTimestamps proof over the manifest digest
  ledger_anchor.json   immudb anchor record for the sealed digest
  artifacts/
    exports/...        exported evidence files
    index/artifact_index.json

Sealing: canonical bytes, one digest, two signatures¶

Signing JSON safely requires a byte-stable representation. The exporter canonicalizes the manifest with JCS-style canonical JSON (RFC 8785): keys sorted by codepoint, no undefined values, no non-finite numbers. Fields that can only be filled after signing, such as the manifest hash itself and the ledger anchor value hash, are blanked before digesting so the committed bytes are well defined. The SHA-256 of those canonical bytes is the manifest digest, and every downstream commitment binds to it.

Two signatures over that digest are required, produced as a detached ES256 JWS. The service environment key (SEK) asserts which KLA environment sealed the bundle. The tenant evidence key (TEK) binds the seal to one tenant; its key ID carries the tenant identifier. The exporter enforces that the two keys are distinct, and in production both live in HashiCorp Vault Transit as non-exportable ECDSA P-256 keys, so signing happens inside Vault and private material never reaches the exporter process.

Artifact integrity uses a dedicated Merkle construction, kla-merkle-v1. Each leaf hashes a domain-separation byte, the artifact path, and the artifact digest; the leaf list is sorted by path bytes; inner nodes hash a distinct prefix over their children. The manifest seals the root, and the bundle carries a per-artifact inclusion proof, so a reviewer can confirm a single file belongs to the sealed set without rehashing the whole archive.

Governance receipts arrive in the bundle already signed. During a governed run, each step seals an ed25519-signed receipt whose content includes prevReceiptHash, the content address of the prior receipt. The hash lives inside the signed payload, so the signature covers the chain link itself. Audit entries carry a second, application-level hash chain: each record stores the hash of its predecessor, computed at write time against the tenant’s latest ledger hash.

{
  "bundleSpec": "evidence-room-bundle-v1",
  "manifestDigestSha256": "4b7f0e0d2b9a51c6f3e8d1a07c5b2e94a6d803f1c2e75b09d4a1f6c8e3b25a70",
  "merkleRootSha256": "a1c9f2d84e07b6531f8c0d2ae95b47c6d310e8f2a7c54b90e6d1a3f8c2b7e415",
  "artifactCount": 214,
  "requiredSigners": [
    "sek",
    "tek"
  ],
  "timestamping": {
    "type": "opentimestamps-bitcoin-mainnet",
    "receiptPath": "timestamp.ots"
  },
  "ledger": {
    "type": "immudb",
    "anchorPath": "ledger_anchor.json"
  }
}

Anchoring the seal outside the system¶

A signature proves who sealed the manifest. Anchors bound when, and register the seal in systems the exporter does not control after the fact.

The first anchor is an OpenTimestamps proof over the manifest digest, committed toward the Bitcoin blockchain. Timestamping is fail-closed: an export with timestamping disabled or a failed stamp refuses to seal, and a deployment can additionally require the proof to be Bitcoin-attested before the export completes.

The second anchor writes the manifest digest back into the immudb ledger under a key derived from the tenant and export IDs, using a verifiable write. The anchor record in the bundle carries the transaction ID and the server’s transaction proof, and the offline verifier later checks that the proof actually binds that key and value to that transaction. If the anchor write fails, the export fails.

Anchor failures and evidence failures never mask an executed action. A degraded write path is recorded as degraded, and a sealed bundle that could not obtain a required anchor is an error state with a named reason.

Collection under load: pagination as a verified protocol¶

The immudb server caps a prefix scan at 1,000 entries per page. A naive unpaged scan of a large tenant ledger silently truncates, which is the worst possible failure for evidence: a plausible bundle missing its tail. The collector therefore treats pagination as a protocol with invariants, and every invariant violation aborts the export with an IncompleteLedgerCoverageError.

Each page is requested with an explicit seek key, the last key delivered by the previous page. Pages stream through a handler as they arrive; nothing accumulates unboundedly in memory. A scan terminates only when a page comes back short of the limit or the configured end of range is passed, and every page is checked before its rows count as covered.

For broad, high-volume scans, candidate record bodies spill to disk as newline-delimited JSON during the scan and are rehydrated later in bounded batches under a byte-weighted budget, holding at most one maximum-sized page of bodies in memory at a time. Fixed budgets bound reads, bytes, concurrency, and wall-clock time, including an idle deadline that a verified advancing page resets and an absolute deadline that nothing resets. Hitting a budget is a refusal to seal, with the budget named in the error.

• Oversized page: a page with more rows than requested aborts the scan.
• Keyless row: every row must carry the key that pagination depends on.
• Out-of-prefix key: a returned key outside the scanned prefix aborts.
• Regressing key: keys must strictly advance in byte order across pages.
• Duplicate key: a key seen twice in one scan aborts.
• Non-advancing cursor: an incomplete page whose last key fails to move the seek position aborts, closing the infinite-loop and silent-truncation cases at once.

Asserting completeness at the end¶

Surviving the scan is necessary and insufficient. When a scoped collection finishes, the collector asserts that every requested selector (execution IDs, lineage record IDs, decision request IDs, policy IDs, frameworks, controls) was actually satisfied by collected entries, and aborts with the missing selectors listed if any were not.

Each collected entry must also carry a trusted attestation: it came from the immudb source, the server response verified against locally maintained trust state, and the inclusion proof verified against the response root. Entries that fail that predicate are excluded and recorded as omissions with a stable reason code rather than exported as if trustworthy. The manifest then reports coverage per section as complete, partial, missing, or not applicable, with reasons for anything short of complete.

Two structural checks close the cases pagination cannot see. The audit hash graph must be connected: because concurrent writers can race the latest-hash pointer, the ledger forms a directed acyclic graph rather than a strict line, and the verifier requires exactly one root; a second root means a member is missing from the export or a second genesis exists. Receipt chains verify their prevReceiptHash links from genesis to last; a chain cannot detect truncation at its end from the links alone, so completeness there is the caller’s contract, and the verifier accepts an expected terminal hash to close that final gap when the caller has one.

Independent verification: what an auditor actually runs¶

The @kla/evidence-verifier package ships a CLI that takes a bundle directory and reruns the integrity checks offline, from the bundle contents alone, with no KLA service in the loop. Each check reports pass or fail with its own errors, and the process exits non-zero on any failure, so the run drops directly into a script or a CI job. A JSON mode emits the full machine-readable result. An optional check sweeps the bundle for caller-supplied forbidden markers, reporting matches only as hash prefixes.

$ evidence-verifier ./export-2026-08 --json > result.json
$ evidence-verifier ./export-2026-08
Evidence bundle: ./export-2026-08
PASS  manifest-signature    SEK and TEK ES256 signatures verify over the manifest digest
PASS  receipt-signatures    41 receipt chains verify from genesis to last
PASS  ledger-hash-chain     1,842 ledger records recompute; hash graph resolves to one root
PASS  merkle-inclusion      214 artifacts match their digests and inclusion proofs
PASS  ots-anchor            timestamp.ots commits to the manifest digest
Result: PASS (5/5 checks passed)

The stated limits of offline proof¶

An honest verification story names its boundary, and the verifier’s boundary is stated in its documentation and in the published tamper-proof evidence page.

Offline verification proves internal consistency. The public keys travel inside the bundle, so binding them to KLA requires pinning them out-of-band against a KLA-published key set; without that step, a from-scratch forgery with fresh keys would self-verify. The Bitcoin attestation inside the timestamp proof carries a block height that offline checking accepts as a claim; confirming it, or upgrading a pending calendar receipt, needs the live chain. The immudb anchor proof binds the sealed digest to one ledger transaction, and confirming that transaction’s inclusion against the ledger’s signed state needs a networked check.

These limits are check-shaped rather than vague: each one corresponds to a specific verification a motivated auditor can perform with public infrastructure, and none of them weakens what the offline run does prove about the bytes in hand.

Negative paths: the tamper suite¶

The verifier’s test suite constructs a valid fixture bundle and then breaks it one byte at a time, asserting that each surface turns exactly the right check red: a flipped byte in an evidence file fails merkle-inclusion; in a receipt signature or receipt body, receipt-signatures; in a ledger record or its stored hash, ledger-hash-chain; in the sealed Merkle root or stated digest, the corresponding seal check; in the timestamp proof, ots-anchor.

Signature-shape attacks are covered alongside content tampering: a malleable high-S variant of a valid ECDSA signature is rejected, as is a valid RSA signature presented under an ES256 header, and any signature that is not exactly the 64-byte P-1363 encoding. Fail-closed behavior gets its own suite: a missing timestamp proof, missing keys, a missing ledger anchor, a receipt signed by a revoked key, and an artifact path that escapes the bundle directory all fail, and the escaping path is refused without reading the outside file.

On the collection side, the exporter’s tests drive the pager past the server scan limit, replay duplicate and regressing keys, force byte budgets over their ceilings, and assert that scoped collections fail closed when a range cannot establish complete pagination. The completeness machinery is exercised as behavior under test, on the same code paths production exports run.