AI Agent Runtime Governance Capability Test Suite

Procedure

1. 01Invoke the consequential tool call through the intended governed route with a policy that blocks it; confirm the business system did not change state.
2. 02Attempt the same action through every alternate route the architecture admits: direct API calls, secondary integrations, operator consoles.
3. 03Ask the vendor to name the exact component that enforces each route and the team that owns its configuration.

Expected of any governed platform

A block or require_approval outcome prevents the side effect on the governed route, and the vendor can enumerate which routes are governed and which are unprotected.

Observed KLA behavior

On the gateway path, the governance gateway seals a decision receipt to the evidence ledger before the tool executes; block and require_approval terminate the call and the tool executor is never invoked. Tool policy is additionally checked in the tool hub on every MCP invocation. Run admission at the execution API is authenticated and role-checked without a policy decision; per-action policy enforcement happens at the tool boundary inside the run.

Evidence

• services/governance-pep/src/gateway.ts: decision sealed before the side effect; block and require_approval short-circuit
• governance-gateway.test.ts: 15 cases on the gateway decision path
• cmb-aml-governance-gateway.test.ts: 16 end-to-end cases on a banking workflow
• scripts/check-production-governance-gateway.mjs: deployment guard that the gateway is enabled in production

Qualifications

• The gateway is enabled by deployment configuration; the shipped production and dev overlays enable it, and a self-hosted operator must do the same.
• The internal connector-execution endpoint trusts that the tool gate ran upstream; it authorizes the caller against the bound connection and performs no second policy evaluation.

Procedure

1. 01Take the policy decision service offline, or inject a transport failure, while the agent proposes a consequential action.
2. 02Repeat with an internal policy-engine failure: no policy resolved, a policy pack that fails signature verification, an unavailable guardrails dependency.
3. 03Record the outcome, the reason code, and whether any configuration can turn a failure into an allow.

Expected of any governed platform

Every failure mode produces block or require_approval with a machine-readable reason. No configuration value can convert an evaluation failure into an allow.

Observed KLA behavior

Fail-closed at every layer found in review: transport failure at the enforcement point, no policy resolved, pack signature failure, unavailable guardrails runtime, entitlement-provider failure, AI-judge outage, and evidence-store failure all deny. The environment override accepts block or require_approval and rejects allow. Production defaults to block; other environments default to require_approval. Publishing a policy pack whose default decision is allow or warn fails compilation.

Evidence

• services/governance-pep/src/decisions.ts + environment.ts: fail-closed decision builder; override cannot be allow
• policy-evaluation-service.test.ts: 12+ fail-closed cases including pack-verification failure and “must not fail open” regression
• services/policy-engine/src/policy-pack/lint.ts: allow/warn default decision is a publish-time error
• transition-gate-decision.test.ts: errored policy outcome blocks and is excluded from human-recoverable reasons

Qualifications

• Environments outside production fail closed to require_approval, so an outage there floods the approval queue while actions stay paused.
• The fail-closed default derives from the runtime environment variable; a production service started with a wrong NODE_ENV degrades to require_approval.

Procedure

1. 01Trigger a require_approval outcome, approve it, then mutate a material parameter (amount, beneficiary, target record) before the run resumes.
2. 02Resubmit the resume payload carrying the original approval and the changed arguments.
3. 03Confirm which side recomputes the binding: the caller’s payload or the server’s sealed record.

Expected of any governed platform

The approval is bound to a canonical digest of the exact arguments, the binding is re-verified server-side at resume from a record the caller cannot supply, and a mismatch blocks the action.

Observed KLA behavior

At approval time the gateway seals a canonical SHA-256 hash of the tool arguments into the durable ledger. At resume it recomputes the hash from the submitted arguments and compares it to the ledger copy; the hash carried on the resume payload is deliberately ignored. A mismatch, and equally an absent sealed hash, blocks with reason tool_args_mismatch.

Evidence

• services/governance-pep/src/gateway.ts: TOCTOU re-verification against the ledger-sealed hash only
• governance-gateway.test.ts: “TOCTOU: Phase-2 resume with mutated arguments fails closed and never executes”
• cmb-aml-governance-gateway.test.ts: “fails closed when the decision input changes after approval”

Qualifications

• The cryptographic re-check runs on the gateway path. Governed connector tools resume through the agent-runtime state object, which pins the arguments structurally without a second hash comparison.

Procedure

1. 01Create an approval with a due time, let it lapse, then attempt to approve it through every decision surface the platform exposes.
2. 02Confirm what authority an expired approval retains and which surfaces enforce the expiry.

Expected of any governed platform

An overdue approval refuses approve and reject on every decision surface, leaving escalation as the only path.

Observed KLA behavior

Approvals carry a due time (default one hour). Both decision surfaces refuse an overdue decision: the control-plane surface, used by the Decision Desk, computes overdue state and allows only escalation, and the execution API’s decision update requires the due time to still be in the future, returning a conflict after the deadline. Maker–checker separation is enforced server-side on both paths.

Evidence

• services/api/src/routers/approvals.ts: overdue approvals accept escalate only; maker cannot check
• services/execution-api/src/routes/approvals.ts: decision update requires due_at in the future
• approval-decision-self-approval.test.ts: post-deadline decision returns 409 without signaling the workflow
• approvals.maker-checker.test.ts: 10 cases

Qualifications

• Tool-gate approvals wait indefinitely by design; the configurable timeout applies to explicit human-approval workflow nodes.

Procedure

1. 01Capture an approved decision, then replay it against a different run, a different tool call, a different tenant, and the same call with different output.
2. 02Confirm the scoping key of the stored decision.

Expected of any governed platform

Decisions and approvals are scoped to tenant, run, and tool call; no replay crosses any of those boundaries.

Observed KLA behavior

The idempotency key is tenant:execution:gate, where the input gate embeds the tool-call id and the output gate additionally embeds the output hash. A replayed committed call returns the stored disposition; a key never authorizes work under another tenant or execution. Fail-closed approvals derive a deterministic approval id from the run and gate, so retries reference one approval.

Evidence

• services/governance-pep/src/idempotency.ts: key structure
• cmb-aml-governance-gateway.test.ts: “does not replay an approval across execution or tenant ledger keys”

Procedure

1. 01Deliver the same governed tool call twice concurrently; deliver it again after a completed run; kill the worker between decision and completion and let the orchestrator retry.
2. 02Count the side effects and inspect the relationship between decisions, executions, and evidence records.

Expected of any governed platform

One side effect per approved action under concurrent and sequential retries, with the crash-window behavior stated precisely.

Observed KLA behavior

A durable write-ahead ledger records intent before execution and commits the result after; a replayed committed call returns the cached result without re-executing, and the losing writer of a concurrent duplicate returns the winner’s result. Every terminal branch, including block and cancel, commits the key so a retry returns the disposition. After a crash between intent and commit, the gateway re-drives once and forwards the idempotency key to the downstream connector.

Evidence

• services/governance-pep/src/gateway.ts: write-ahead intent, committed short-circuit, one-shot output release
• governance-gateway.test.ts: “exactly-once: a replayed committed call returns the cached result without re-executing”; concurrent-loser case
• cmb-aml-governance-gateway.test.ts: re-drive across an orchestrator retry without a second side effect

Qualifications

• In the crash window, exactly-once depends on the downstream system honouring the forwarded idempotency key; a downstream that ignores it degrades to at-least-once. This is stated in the source.
• The durable ledger table relies on tenant-prefixed keys for isolation; it has no row-level-security policy yet.

Procedure

1. 01Trace where connector credentials are resolved and which process memory they enter during a governed tool call.
2. 02Attempt server-side request forgery through a connector URL that resolves to internal or metadata addresses.
3. 03Submit write statements through a read-only database connector.

Expected of any governed platform

Credentials resolve inside the control plane only; egress is pinned to validated addresses; read-only connectors refuse writes at more than one layer.

Observed KLA behavior

Connector credentials resolve in the control plane and are materialized only into outbound request headers or a database client; the execution worker sends the tool input and receives the result. Connector egress validates every resolved address at connect time, pins the connection to the validated IP, keeps TLS names on the original host, and refuses redirects. Database reads pass a keyword guard with literal masking, then run inside a database-enforced read-only transaction under the connection’s least-privilege role. Secret-shaped values are rejected from durable installation records at the API boundary.

Evidence

• services/api/src/services/connector-execution.ts: control-plane custody; DNS-pinned egress; BEGIN READ ONLY
• connector-network-safety.ts: metadata, link-local, multicast, and documentation ranges blocked; private ranges gated by explicit configuration
• mcp-installation-secret-safety.ts: secret-pattern rejection at the API boundary

Qualifications

• Locally spawned MCP tool servers inherit the worker process environment; a hostile MCP server binary could read variables present on that pod.
• Credential custody is architectural; there is no automated negative test asserting that a model prompt can never contain a credential.
• A connector can be configured to skip TLS verification; that switch is part of the connection record a reviewer should check.

Procedure

1. 01Alter one byte of a stored decision record, an approval audit record, and an evidence receipt, through whatever privileged access the platform’s storage admits.
2. 02Read each altered record through the product and export it; record where detection fires.

Expected of any governed platform

Alteration of any governance record is detected on read or on export, through integrity mechanisms independent of the mutated store.

Observed KLA behavior

Governance records append to an immutable ledger through verified writes that bind key and value to the transaction proof. Audit-trail and policy-gate reads go through verified reads that recompute the record’s content hash and its inclusion proof; a mismatch refuses to serve the record. Decision receipts are signed with Ed25519 over a canonical serialization and chained: each receipt embeds the hash of its predecessor inside the signed body, so edits, deletions, and reordering break the chain at a named index. The relational transition log is append-only under a database trigger and carries the same chain hash.

Evidence

• services/api/src/services/immudb-multi-tenant.ts: hash recomputation and trusted-read verification on audit and policy-gate reads
• services/governance-pep/src/receipt-chain.ts + signing.ts: signed hash chain; 28 test cases including tamper, reorder, truncation
• export-api.receipt-ledger-bundle.test.ts: tampered receipt and tampered ledger record turn the export red

Qualifications

• Some policy-decision reads check a content hash stored alongside the record without recomputing a server inclusion proof, and display read models such as the control-decision table are mutable rows; alteration on those paths is established by comparison with the sealed ledger and at export.
• Detection fires when a record is read or exported; there is no continuous background re-verification job.
• End-truncation of a receipt chain is detected only when the verifier is given an independently stored terminal hash.

Procedure

1. 01Export a Sealed Evidence Bundle for a governed run, move it to a machine with no network access, and run the published verifier.
2. 02Flip one byte in each artifact class and re-run; every flip must turn the run red with a named check.

Expected of any governed platform

A self-contained verifier proves signatures, hash chains, and inclusion proofs from the bundle alone, states clearly what it cannot prove offline, and fails closed on tampering.

Observed KLA behavior

The evidence verifier runs five checks with no network access, using the key set carried in the bundle: manifest signature under service and tenant keys, receipt signature chains using the runtime verifier, ledger hash-chain recomputation, Merkle inclusion against the retained transaction proof, and timestamp-anchor consistency. One-byte tampering in any artifact class turns the corresponding check red in the automated suite, including signature malleability and wrong-algorithm cases. The exit code is the verdict.

Evidence

• packages/evidence-verifier: five checks, command-line verifier, ~55 automated cases
• verifier.test.ts: one-byte tamper per artifact class; revoked key; path escape; malformed key set

Qualifications

• The verifier’s key set travels inside the bundle, so a passing run proves internal consistency of the bundle as exported; detecting a re-signed bundle from an untrusted exporter requires comparing the bundle’s keys against independently received key material, and built-in key pinning is a roadmap item.
• Merkle inclusion is checked against the transaction proof retained in the bundle; verification against the ledger’s independently signed state is a roadmap item, and confirming the timestamp anchor on the public chain requires a networked step.
• A bundle that declares only unsigned receipts passes the receipt check with zero verified chains; the verifier reports the unsigned count and an auditor must read it.