Runtime AI Governance: Before, At, and After

A precise definition of runtime AI governance¶

Runtime AI governance is the continuous enforcement and evidence layer that decides whether a proposed AI action is authorized, policy-compliant, and safe to execute in its current context. It applies controls while the system is operating and preserves a durable record of the decision and its outcome.

The unit of governance is the consequential action. A model response can be informative, while a tool call can move money, update a customer record, publish content, change an entitlement, submit a regulatory filing, or trigger another system. The action carries the business consequence, so it receives the policy decision, human authority, and evidence record.

Runtime governance also covers the operating context around that action: which agent and principal initiated it, which tenant and environment it belongs to, which tool and destination it targets, which policy and version apply, which thresholds were evaluated, and which human role owns an exception. That context turns a general rule into an enforceable decision.

Why the category is taking shape now¶

Agentic systems combine model output with tools, memory, permissions, and repeated execution. The OWASP Excessive Agency guidance identifies excessive functionality, permissions, and autonomy as root causes of damaging agent actions. Its mitigations include narrow tools, narrow permissions, and independent approval for high-impact actions.

Vendors are also naming the execution layer directly. Holistic AI introduced runtime agentic enforcement for tool calls, access, and cost controls in April 2026. These sources use different product and framework vocabularies. They point to the same engineering boundary: the moment a proposed action reaches a system that can create a real side effect.

Broader governance frameworks already expect continuous lifecycle work. The NIST AI RMF Core treats governance as cross-cutting and calls for regular monitoring and improvement across deployed systems. Runtime governance supplies one technical mechanism for implementing that intent at the action boundary.

Before execution: define authority and control intent¶

A runtime decision can only be as clear as the control model supplied to it. Before an agent runs, the organization defines who owns the system, what the agent may do, which tools and data it may reach, where human authority begins, and what evidence each decision must produce.

This is where governance-of-record systems and management frameworks do essential work. ISO/IEC 42001 defines an AI management system for establishing policies, objectives, processes, risk management, and continual improvement. Runtime controls consume that institutional intent as executable policy and operating configuration.

• Identity and ownership. Give each agent and principal a stable identity, an accountable owner, a purpose, and an approved environment.
• Delegated authority. Define permitted actions, destinations, values, rates, data boundaries, time windows, and delegation depth.
• Policy and risk bands. Convert control intent into explicit conditions, decisions, reason codes, approval groups, and escalation deadlines.
• Tool and data scope. Grant the minimum functions and records required for the job, with separate read and write authority.
• Human decision design. Name the accountable role, maker-checker rule, context package, response window, and timeout behavior.
• Evidence contract. Specify the identifiers, policy version, action inputs, decision basis, human activity, execution result, retention, and integrity proof to preserve.

At decision time: enforce the policy on the proposed action¶

The runtime checkpoint receives a proposed action before its side effect commits. It resolves the agent and principal identities, loads current authority and policy, evaluates action context, and returns a binding outcome.

Four outcomes give teams enough range to keep routine work flowing and reserve human attention for consequential exceptions. Precedence matters when several rules match: a stronger decision must survive aggregation.

A runtime checkpoint needs fail-closed behavior¶

A policy decision has value only when the action path obeys it. The checkpoint must sit on the route to the tool or downstream system, and the execution path must stop when the policy engine, evidence store, identity source, or approval authority is unavailable.

A high-integrity design seals the decision basis before the side effect, uses an idempotency key so retries do not duplicate the action, and binds a later approval to the same action arguments. Output controls can also hold a result before release when the tool has already run. These controls address policy bypass, approval replay, argument substitution, and duplicate execution.

• One mediated path from the agent runtime to consequential tools
• Current identity, authority, policy, and destination context on every evaluation
• A deny or safe hold when required dependencies cannot produce a trustworthy decision
• Durable human waits with explicit approval identity and action binding
• Idempotent execution across retries and resumable workflows
• Decision evidence written before commitment and outcome evidence written after

After execution: preserve proof and improve the controls¶

Runtime governance continues after the decision. The operating record should connect the proposed action, policy evaluation, matched rules, human decision, downstream result, timestamps, versions, and integrity evidence under stable execution and decision identifiers.

That record supports several jobs. Operators investigate an individual run through execution lineage. Risk teams measure blocked actions, warnings, approval load, overrides, latency, and policy effectiveness. Auditors sample decisions and verify evidence. Control owners update policy when outcomes reveal drift, recurring exceptions, or gaps in coverage.

The EU AI Act gives this loop regulatory relevance for high-risk systems. Article 9 requires a continuous, iterative risk management system. Article 12 requires technical logging capabilities. Article 14 requires effective human oversight during use. Runtime evidence can support those duties when the system scope, provider or deployer role, and applicable obligations are established for the specific use case.

Observability, guardrails, governance of record, and runtime governance¶

These four layers solve different problems and often share data. Teams need the distinction because a single vendor may offer several layers under one product name.

OpenTelemetry defines observability signals such as traces, metrics, logs, and baggage. Those signals describe system activity and health. Runtime governance can consume them as context and emit policy decisions back into the same telemetry pipeline.

Where each layer reaches its limit¶

Layer boundaries become clear when a system fails. Each layer has a characteristic question it can answer and a gap that another layer must cover.

• Observability reaches its limit at authority. It can show that an agent called a payment API, which route handled the request, and how long it took. A separate policy decision establishes whether that agent had authority to make this payment for this principal under current limits.
• Guardrails reach their limit at institutional context. A toxicity filter can classify an outbound message. A runtime policy also needs customer consent, jurisdiction, agent role, communication channel, case state, and approval authority.
• Governance of record reaches its limit at enforcement. A risk assessment and control mapping establish the expected rule. The execution path needs a machine-readable version of that rule at the moment of action.
• Runtime governance reaches its limit at coverage and control quality. The layer only governs actions routed through it, using the identities, policies, and evidence contracts supplied to it. Inventory, integration testing, policy review, and assurance remain essential.

The minimum architecture for runtime AI governance¶

A category-level implementation needs six connected capabilities. Product packaging can vary; the control responsibilities stay stable.

An AI control plane provides these capabilities across agent runtimes, tools, and workflows. It centralizes policy, approval, and evidence contracts while each enforcement checkpoint stays on the action path.

• Agent and principal identity. Resolve the acting agent, the human or service principal behind it, tenant, environment, and current roles.
• Authority and least privilege. Bind actions to explicit tools, resources, destinations, values, rates, and time limits.
• Contextual policy evaluation. Evaluate policy against the proposed action and current business state with versioned rules and reason codes.
• Human decision routing. Pause selected actions, route them to the accountable reviewer, enforce maker-checker rules, and bind the result to the original request.
• Execution mediation. Place the decision on the actual route to the tool, preserve idempotency, and enforce the returned outcome.
• Lineage and evidence. Record decision and outcome data with integrity controls, retention, export, and independent verification paths.

How KLA applies the category¶

The KLA Control Plane applies this model to governed actions in regulated Processes. KLA Runtime is the execution layer. The application-layer KLA Policy Engine evaluates the governed action and returns allow, warn, require_approval, or block. A require_approval outcome creates a Decision Request for the accountable reviewer in Decision Desk.

The operating record remains connected after the decision. Lineage Explorer presents the execution chronology. Evidence Room carries review artifacts and Sealed Evidence Bundles. The policy-as-code checkpoint, human oversight, live intercept, and execution lineage concept pages show each part of the KLA Runtime model.

This architecture supports govern-in-place integration. Existing agent frameworks and workflow engines can send governed actions through the checkpoint while retaining their execution responsibilities. The policy, approval, and evidence contracts remain stable across the integrated path.

A practical evaluation checklist¶

Use these questions when evaluating a runtime governance design, product, or control claim.

• Which exact actions are mediated, and which paths can reach a consequential tool outside the checkpoint?
• How are agent, principal, tenant, environment, destination, and delegated authority resolved?
• What are the complete decision outcomes, their precedence, and their failure behavior?
• Can a human approval be replayed for different arguments, tools, principals, or policy versions?
• What happens when policy, identity, evidence, or approval dependencies are unavailable?
• How does the system prevent duplicate side effects during retries and workflow resumes?
• Which decision fields are sealed before execution, and which outcome fields are added afterward?
• Can an independent reviewer verify the evidence and trace it to policy and human authority?
• How are warnings, blocks, overrides, approval load, drift, and control effectiveness reviewed?
• Which standards or legal duties apply to this system, use case, jurisdiction, and organizational role?