Deterministic replay of an AI agent decision means re-running the deterministic part of the decision, the policy rules, over the inputs recorded at the time, under the policy version that was in force, and getting the same outcome. Model output, AI-judge verdicts, tool results and human approvals cannot be regenerated reliably, so they are recorded once and read back during replay. Side-effecting tool calls are never repeated. A replay that matches the record shows that the recorded decision follows from the recorded inputs and policy. It does not show that an external system applied the action; that proof comes from the external system.
This post separates three meanings of replay, lists what has to be recorded for replay to work, walks through one synthetic replay, and describes what KLA replays today. Replay depends on knowing what the agent read: how to prove what an AI agent read covers that record.
Three things people mean by replay
Teams use the word for three different operations. Name the one you mean in the audit plan, because each proves something different.
| Operation | What runs | Deterministic | Side effects | What it shows |
|---|---|---|---|---|
| Playback | Nothing. A viewer steps through the recorded events in order | Yes, it reads a fixed record | None | What the record says happened, in sequence |
| Decision replay | The policy rules, over recorded inputs, under a pinned version | Yes, when the rules and inputs are deterministic | None. No tools, approvals or evidence records | That the recorded decision follows from the recorded inputs and policy, or where it diverges |
| Re-execution | The agent again: model calls, retrieval and tools | No. Model output and source data change between runs | Yes, unless every tool is stubbed | How the current system behaves on a similar input. It says little about the original run |
What replays deterministically and what does not
A policy rule that compares an amount with a threshold, checks a role or tests a field against a list gives the same answer for the same inputs. That makes it replayable, provided the inputs and the exact rule version are on record.
A hosted language model can return different text for the same prompt across calls and model updates. Lowering sampling temperature narrows the variation without removing it, and some current models do not accept a temperature setting. Treat model output as a recorded input: store what the model returned and replay the decision from that.
An AI judge, a model asked to classify or compare text inside a policy check, has the same problem. Record its verdict with the provider, model, prompt version and hashes of what it compared. A replay can then check the rest of the decision around that recorded verdict.
External systems move on. A customer record, a sanctions list or an account balance read at decision time will differ a week later. Replay from the recorded read, with its source version, and never from a fresh query.
- Deterministic: threshold, role, list-membership and schema checks; outcome precedence among
allow,warn,require_approvalandblock; routing to a reviewer role. - Recorded and reused: model output, AI-judge verdicts, retrieval results, tool results, human decisions, timestamps used by rules.
- Never repeated: writes, payments, messages, filings and any other call with an external effect.
What to record so replay works
Most failed replays trace back to one missing item. The table lists what a decision replay needs, with example field names.
| Item | Example fields | Replay fails without it because |
|---|---|---|
| Policy identity and version | policy_id, policy_version, policy_pack_hash | The rule set in force cannot be selected or checked against the archive |
| Decision inputs | The context the rules read: action, amount, resource, actor roles, tool arguments, with masked fields marked | Rules have nothing to evaluate; an inputs digest alone cannot be re-evaluated |
| Tool input schema version | tool_id, tool_version | Stored arguments may no longer fit the current schema |
| Recorded model and judge output | Model output text or fields; judge verdict, model, prompt_version, operand hashes | The decision would depend on a new model call |
| Reference data versions | Sanctions list version, rate table version, as-of times of reads | A rule that reads reference data gives a different answer today |
| Agent configuration | Agent manifest hash resolving to model, instructions and tools | The reviewer cannot tell which agent release produced the inputs |
| Recorded outcome | decision, matched_rules, reason_codes, occurred_at | There is nothing to compare the replay with |
Worked example: replaying one synthetic credit decision
A fictional bank’s credit agent requested a disposition of EUR 24,000 for a self-employed applicant. Policy credit-disposition-policy 4.2.1 returned require_approval under rule manual-review-above-20000. The auditor replays the decision twice: under the recorded version, and under a draft 4.3.0 that adds a block for self-employed applicants above EUR 15,000. All identifiers and values are invented, and the digests are shortened.
{
"run_id": "run_syn_0418",
"replay_scope": "deterministic-only",
"recorded": {
"policy_id": "credit-disposition-policy",
"policy_version": "4.2.1",
"policy_pack_hash": "sha256:a4fc…278a",
"decision": "require_approval",
"matched_rules": ["manual-review-above-20000"],
"occurred_at": "2026-10-02T10:02:11.204Z"
},
"replayed": [
{
"policy_version": "4.2.1",
"decision": "require_approval",
"matched_rules": ["manual-review-above-20000"],
"delta": "none"
},
{
"policy_version": "4.3.0-draft",
"decision": "block",
"matched_rules": ["block-self-employed-above-15000"],
"delta": "require_approval -> block"
}
],
"not_replayed": [
{
"gate": "memo-consistency-check",
"reason": "needs_original_text"
}
]
}- Recorded version: replay under 4.2.1 returns
require_approvalwith the same rule. The recorded decision follows from the recorded inputs. - Draft version: replay under 4.3.0 returns
block. The delta shows the impact of the draft on this run before it ships. - Not replayed: one gate used an AI judge on memo text that was not stored. The replay reports it under
needs_original_textand leaves that gate out of the comparison. - Not shown: whether core banking recorded the disposition. The tool receipt and the core-banking record answer that.
A replayed decision does not prove an external action
A matching replay shows that the policy, applied to the recorded inputs, gives the recorded outcome. It does not show that the downstream system accepted the call, applied the change, applied it once, or kept it. A payment can be rejected after the agent’s call returned. A record can be changed by another process a minute later.
Proof of an external effect comes from the external system: a downstream receipt, an after-state digest, an effect reference the system of record can resolve, or a reconciliation extract. The AI agent execution record shows these as tool_calls[].result_digest and downstream_effects[] with before and after state digests. An auditor tests them separately from the replay.
Replay must also never create a second effect. An idempotency key per action lets a retried or resumed call return the stored result. When a call was sent and no result was recorded, the outcome is unknown: the right response is reconciliation with the destination system, and a replay cannot settle it.
What we learned building decision replay
We first offered policy simulation through hand-built test cases. Dogfooding the product in October 2026 made the problem plain: a compliance officer will not write a test case with dozens of fields to check a policy change. The question they ask is what the change would have done to recent decisions, for example how many more cases would go to human review. That is the question our impact replay answers, by running a draft over recent recorded runs and listing each run whose outcome would change.
Replay only works on inputs recorded from the start. We began storing the evaluated decision context on each decision record on 23 September 2026. Decisions recorded before that date cannot be replayed; impact replay counts them under no_recorded_context and never counts them as matches.
Compare matched rules as well as outcomes. For a held-payment release showcase we prepared for an analyst briefing in October 2026, on synthetic data, we ran five policy simulations. Each result recorded the expected and actual outcome, the matched rule IDs, the policy ID and version, and an evidence ID, and all five matched. The block result for a confirmed sanctions match listed two matched rules, the sanctions stop and the general release-approval rule, with one reason code. An outcome-only comparison would not show that both rules fired. An earlier simulation in the same work failed closed because its tool input lacked trusted validation; we rewrote the rule to read Process work-item fields, and the failed run stays in the audit history.
Count reason codes before explaining a rate. In an internal benchmark of an AI-judge check in September 2026, one policy variant appeared to intervene wrongly on 98% of cases. The reason codes showed that those interventions were 565 judge_unavailable results caused by input truncation, identical across models. We had first blamed the match logic. A later variant that compared only the instruction with the current action had no truncation.
How KLA replays decisions today
KLA records each policy decision with the policy ID, version, policy pack hash, matched rules, reason codes, outcome and time, and keeps a redacted copy of the decision inputs for replay. Decision records written before KLA kept that copy are reported as not replayable.
Per-run replay. KLA re-evaluates the recorded decision inputs of one run with AI-judge and guardrail checks switched off and no approval creation. The response is labelled deterministic-only and lists, per gate, the previous and new decision.
Impact replay. A Simulation re-evaluates a chosen policy version, usually a draft, over the last 50, 200 or 1,000 recorded runs in the policy’s scope (200 by default) and lists each run where the recorded and draft outcomes differ. It reads decision records only and creates no evidence, approvals or decision records. Every run it cannot replay is counted under one reason: no_recorded_context, input_not_stored, tool_schema_changed, decided_by_safeguard or needs_original_text.
AI-judge checks. KLA records the provider, model, prompt version, hashes of the compared operands, verdict, confidence and method for each judge check. Replay does not call the judge again; runs that depend on one are reported under needs_original_text.
Agent configuration. Each execution references an agent manifest hash, and the manifest history keeps the full manifest: model provider and name, instructions and tools.
Side effects. The runtime keys each governed tool action by tenant, execution and gate. A retry returns the committed result. A call with a recorded intent and no result stops with "execution outcome is unknown; operator reconciliation is required".
Playback. Lineage Explorer and the agent runtime view show the recorded sequence of a run. They do not re-execute it.
Limits. KLA does not re-run model calls or tools as part of replay. The evidence verifier documentation states that its checks prove recorded decision linkage, "not downstream tool execution".
Replay as an audit procedure
Use decision replay as a reperformance test on a sample. Record every result in the workpaper; the completed AI agent audit workpaper includes a replay row with its limitation stated.
- Select: draw the sample from the reconciled population, including
blockandrequire_approvaloutcomes and runs around policy changes. - Pin: resolve the recorded policy version and pack hash against the release archive.
- Replay: re-run the decision over the recorded inputs with no tool access.
- Compare: match outcome and rules with the record. Classify each difference: rule change, input not stored, schema change, judge or model dependence, or an unexplained divergence.
- Test effects separately: trace each executed action to the downstream system and reconcile it.
- Report limits: state which gates were left out and why, and do not extend the replay conclusion to external effects.
Frequently Asked Questions
What are AI agent audit log best practices for deterministic replay?
Record the policy ID, version and pack hash; the decision inputs the rules read; tool schema versions; model and AI-judge output; reference data versions; the agent configuration hash; and the outcome with matched rules. Write the record when the decision is made, in append-only storage, and keep enough content to re-evaluate it.
Do immutable audit logs make AI agent decisions replayable?
Immutable storage shows that a record has not changed since it was written. Replay also needs the record to contain the decision inputs and a resolvable policy version. A tamper-evident log of tool names and timestamps is intact and still cannot be replayed.
How can I audit and replay AI agent decisions?
Sample decisions from a reconciled population, resolve the recorded policy version, re-run the policy over the recorded inputs without tool access, and compare the outcome and matched rules with the record. Test executed actions separately against the downstream system, because a matching replay does not prove an external effect.
Can you replay an LLM decision deterministically?
Not by calling the model again. Hosted model output varies between calls and versions. Record the output the agent used and replay the deterministic checks around it.
Key Takeaways
Deterministic replay covers the policy decision: recorded inputs, a pinned policy version and recorded model and judge output give a result an auditor can reperform. Everything with an external effect stays outside the replay and is proven from the system that holds it. For the record format, use the AI agent audit log schema and the policy decision schema. For how KLA connects decisions, tool calls and effects in one record, see Execution Lineage.