AI Agent Audit Trails: LLM Observability vs Audit-Grade Evidence

What LLM observability records¶

Many LLM observability tools emit or ingest OpenTelemetry data. The OpenTelemetry semantic conventions for generative AI define spans for inference, embeddings, retrieval, memory and tool execution, with attributes such as gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.tool.name. The conventions carry Development status, so names can still change.

Content is Opt-In in those conventions. Model input and output (gen_ai.input.messages, gen_ai.output.messages), tool arguments and results (gen_ai.tool.call.arguments, gen_ai.tool.call.result) and retrieved documents (gen_ai.retrieval.documents) are recorded only when someone enables them. A trace with default settings can show that a tool ran and how long it took, with no record of what it was asked to do or what it returned.

This data serves engineering well: finding a failed step, a slow retrieval or an expensive prompt, and comparing prompt versions. Engineering use also explains the usual storage choices: sampling to control volume, retention set by storage cost, and stores that operators can edit or purge.

We met the gap between the two records in our own product. In a merchant-onboarding proof of concept in September, the case history linked each step to Lineage Explorer, and the links showed "Trace not found" for a run paused at a sign-off. The run published its trace spans only when it finished, while the gate receipt for the pending decision already existed. We changed Lineage Explorer to read the steps the run had recorded so far from the execution record when no trace has been published, and to say that the run has not ended.

What an audit-grade trail adds¶

An auditor, regulator or legal team reviewing one agent action asks questions a trace was not designed to answer. The table maps each question to a typical observability field and to the field that answers it in the vendor-neutral AI agent audit log schema.

One synthetic action, recorded both ways¶

A fictional bank’s credit agent records a EUR 24,000 credit disposition. The observability trace for the tool step looks like this. Values are synthetic.

{
  "name": "execute_tool set_disposition",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "start_time": "2026-07-21T09:14:31.944Z",
  "end_time": "2026-07-21T09:14:32.433Z",
  "attributes": {
    "gen_ai.operation.name": "execute_tool",
    "gen_ai.tool.name": "set_disposition",
    "gen_ai.tool.call.id": "call_01JZ8V4R1E7P3M9C5Q2T6N8D0A",
    "gen_ai.agent.name": "Credit review agent"
  },
  "status": "OK"
}

The audit event for the same action¶

The audit event for the same call is the public synthetic complete execution record. It carries the same trace_id and span_id, so the two records join. It adds what the span lacks: the delegating user and accountable owner, the policy credit-disposition-policy 4.2.1 returning require_approval under rule manual-review-above-20000, the senior underwriter’s approval at 09:14:31.902Z before the call at 09:14:31.944Z, the arguments and result digests, the before and after state digests in core banking, the business outcome, and a record hash with an Ed25519 signature.

An auditor can verify the signature, check that the approval preceded the call, re-run the policy over the recorded inputs and trace the effect reference into core banking. With the span alone, the auditor can confirm that a tool named set_disposition returned without error.

What the EU AI Act asks of logs¶

Article 12(1) of the EU AI Act requires high-risk AI systems to technically allow automatic recording of events over the system’s lifetime. Article 12(2) ties that logging to identifying situations that may create risk or a substantial modification, to post-market monitoring, and to the deployer’s monitoring of operation. Article 12(3) sets minimum fields for remote biometric identification systems only.

Article 19(1) requires providers to keep automatically generated logs under their control for at least six months, unless other Union or national law provides otherwise. Article 26(6) sets the same minimum for deployers. Corrective action sits in Article 20; quality management, including record-keeping procedures, in Article 17.

The Act leaves the field list to the provider for most high-risk systems. For an agent, traceability of a decision usually means the actor, release, policy, approval, tool effect and outcome fields in the table above. Regulation (EU) 2026/1744, published on 24 July 2026, moved the application date for Article 6(2) and Annex III high-risk systems to 2 December 2027 and for Article 6(1) and Annex I systems to 2 August 2028. The Article 12 logging guide covers the duty in detail.

The four layers of an evidence pack¶

An evidence pack bundles the audit trail for one decision or one period so a reviewer can check it outside the producing system. The Evidence Room sample shows one export.

• Decision record: identifier, time, decision type, outcome and risk classification. Every other artifact refers to it.
• Input context: what the agent read, with source references and versions; the model, prompt and policy versions in effect. How to prove what an AI agent read sets out the read record.
• Governance evidence: policy decisions with matched rules, Decision Requests with reviewer and rationale, escalations and overrides.
• Integrity: a manifest of artifacts with SHA-256 digests, signatures, timestamp anchors and chain of custody, so a reviewer can verify the pack offline.

Architecture patterns for audit-grade records¶

These patterns sit on the action path. They need the agent to call tools through a checkpoint that can evaluate policy and record the result.

A field missing when a record is sealed cannot be added later. At the start of September 2026 a code trace of our own platform found that some in-platform decision records and sealed receipts did not name the acting agent. Sealed receipts cannot be backfilled, so every day without the field added records an auditor could not tie to an agent. We added agent identity to decision records and to sealed gate receipts within two days. Decide the field list before the first production record.

• Write at decision time: record the policy decision before the action proceeds and the approval before the tool call. A record assembled afterwards from logs cannot show order reliably.
• Append-only storage: store events where existing entries cannot be edited or deleted, and chain each record to the previous one by hash.
• Sign and seal: sign each record, and seal exports with a manifest and signature that a reviewer can check with published keys.
• Keep content resolvable: a digest proves a match only if the content can be produced later. Keep the source version or a protected copy for the retention period.
• Record what replay needs: decision inputs, the pinned policy version and recorded model output. Deterministic replay for AI agent decisions lists the full set and its limits.
• Join to telemetry: carry trace_id and span_id in the audit event so engineers and auditors can move between the two records.

Building the evidence layer on an observability stack¶

Teams with tracing in place can add the evidence layer in phases.

• Phase 1, define the questions: list the decision types with audit or legal exposure and the questions an auditor will ask about each.
• Phase 2, place checkpoints: route consequential tool calls through a policy checkpoint that records actor, release, policy, approval and effect fields.
• Phase 3, protect integrity: move audit events to append-only storage with hashing, signatures and sealed exports.
• Phase 4, test the record: sample actions, verify signatures, reconcile the population and replay decisions. The AI agent audit checklist and the audit workpaper give the tests.