A repeatable internal audit engagement for planning, fieldwork, sampling, evidence evaluation, findings, and follow-up. The enterprise framework defines the broad 12-domain audit universe. The AI audit checklist supplies quick control questions, and the accountability matrix assigns roles. This page owns the engagement itself: a scoped assurance question, a reconciled population, risk assessment, design and operating-effectiveness tests, reproducible samples, evidence evaluation, reporting, and closure. The worked figures below describe one illustrative loan-origination audit and can be replaced with the reconciled counts from a live engagement.
1. Set the engagement objective and criteria
Write one assurance question before requesting evidence: During the audit period, were consequential agent actions authorized, governed by approved controls, accurately recorded in the business system, and supported by sufficient evidence? The scope statement then fixes the business Process, agent and Release population, environments, period, decision types, materiality, dependencies, exclusions, and criteria. Every fieldwork procedure must resolve one part of that question.
Keep the authority of each criterion explicit. The IIA Three Lines Model assigns risk management to first-line roles, expertise and challenge to second-line roles, and independent and objective assurance to internal audit. The IIA Artificial Intelligence Auditing Framework is published practical guidance that audit teams customize, and the ISACA Artificial Intelligence Audit Toolkit is a commercial practitioner toolkit covering control design and operating effectiveness. NIST AI RMF 1.0 is a final voluntary framework. Applicable law enters the criteria stack only after the organization documents operator role, intended purpose, classification, and jurisdiction.
The program below separates legal obligations, management controls, professional guidance, and KLA recommendations. Its sample sizes and pass criteria are engagement-planning choices. An internal audit opinion addresses the stated criteria for the defined scope and period. It provides no certification of regulatory compliance, system safety, or future performance.
| Planning field | Required entry | Worked engagement entry | Planning acceptance |
|---|---|---|---|
| Assurance objective | One testable question covering authorization, control operation, outcome accuracy, and evidence | During 1 April-30 June 2026, were consequential actions by the production loan-origination agents authorized, preceded by applicable policy, routed for required human decision, accurately written to the loan system, and supported by complete evidence? | Audit sponsor, lead auditor, and accountable Process owner approve the exact wording |
| Business Process and boundary | Start event, end event, included systems, human roles, and dependencies | Application receipt through approve, decline, or refer outcome; includes identity service, retrieval layer, credit-model service, KLA Policy Engine, Decision Desk, loan system, and Evidence Room | Every component able to influence authority, decision, action, or evidence is named |
| Population and period | Agents, Releases, environments, actions, and date range | 8 production agents; 27 Releases; 184,216 action records; production only; 1 April-30 June 2026 | Counts reconcile to deployment, identity, gateway, Lineage Record, and business-system sources |
| Decision types in scope | Consequential allow, deny, escalate, override, state-change, and recovery outcomes | Approve, decline, refer, policy deny, human override, applicant-data retrieval, credit pull, offer issuance, Rollback, and access revocation | Each material decision type has a population owner and test procedure |
| Materiality | Financial, rights, data, operational, and evidence thresholds | All adverse applicant outcomes; all approvals above EUR 25,000; all special-category or identity data access; all external writes; all incidents, near misses, overrides, and evidence-integrity failures | Thresholds are approved before sample selection and cover qualitative impact |
| Criteria stack | Policy, contract, framework, and applicable legal criteria with authority and version | Lending policy LND-04 v6.2; Agent Control Standard ACS-02 v4.1; retention schedule RS-17; vendor contracts; NIST AI RMF as voluntary guidance; EU AI Act provisions only where the documented role and high-risk classification make them applicable | Every criterion is versioned, owned, effective during the period, and mapped to a test |
| Roles and independence | Engagement sponsor, audit lead, subject-matter support, evidence owners, and quality reviewer | Audit committee sponsor; internal audit lead; lending, IAM, model-risk, data, security, and legal subject-matter support; independent audit quality reviewer | Internal audit has no control ownership or management approval role in the audited Process |
| Exclusions and deliverables | Reasoned exclusions, effect on assurance, report, workpapers, and follow-up dates | Development sandboxes and vendor internal model-training operations excluded; report due 31 July 2026; management responses due 14 August; first retest 30 September | Each exclusion has an owner, rationale, risk effect, and stated opinion boundary |
2. Prove population completeness before selecting samples
Population completeness is an audit assertion. Reconcile the Agent Registry to production deployments, non-human identities, model and tool gateways, Lineage Records, Decision Requests, and downstream business events. Freeze the reconciled extracts with query text, filters, extraction time, row count, and hash. Unexplained differences become exceptions or a scope limitation before sampling starts.
NIST AI RMF 1.0 is voluntary and supports maintaining an AI-system inventory under GOVERN 1.6. A fieldwork population needs more detail: every Agent, Release, model, identity, temporary delegation, tool, consequential action, policy result, human decision, incident, and external effect for the period.
Search for shadow agents through model-gateway traffic, cloud service principals, package and platform spend, browser extensions, automation credentials, tool-gateway calls, and business records with AI-generated provenance. Reconcile temporary and delegated identities by creation, expiry, revocation, sponsoring principal, and observed use. A missing identity or execution source can invalidate the completeness assertion for every downstream sample.
| Population component | Authoritative and corroborating sources | Worked count | Reconciliation procedure | Pass criterion |
|---|---|---|---|---|
| Production agents | Agent Registry; deployment manifests; model-gateway client IDs | 8 | Join agent ID, environment, owner, Process, and active dates; investigate every gateway client absent from the registry | All 8 deployed agents appear in all applicable sources; zero unexplained clients |
| Releases and Rollouts | Agent Registry; immutable Release artifacts; deployment controller; change tickets | 27 Releases / 31 Rollouts | Match `agent_release_id`, artifact hash, approval, deployment time, environment, and Rollback history | Every production interval resolves to one approved Release and Rollout |
| Persistent agent and service identities | IAM directory; secrets inventory; deployment manifests; gateway logs | 19 identities | Match owner, entity type, credential, scope, activation, expiry, and last use to an in-scope Agent or dependency | Zero shared, orphaned, expired-in-use, or unowned identities |
| Temporary identities and delegations | Token service; authority snapshots; delegation records; revocation events | 31 temporary identities / 286 delegated Journeys | Reconcile issue, sponsor, purpose, scope, expiry, use, and revocation; test for use before issue or after expiry | 100% resolve to an approved sponsor and bounded purpose; zero out-of-window use |
| Tools and external dependencies | Tool Catalog; IAM grants; egress gateway; vendor inventory; contracts | 14 tools, including 6 write-capable; 4 external dependencies | Compare registered tools and versions with observed endpoints and effective grants; inspect all unknown destinations | Every observed tool and dependency is approved, owned, versioned, and within Data Boundaries |
| Agent action records | Lineage Records; policy events; tool gateway; model gateway | 184,216 | Join `record_id`, `correlation_id`, event time, agent, Release, policy, and tool effect; sequence duplicate and missing IDs | Counts and keys reconcile; zero unexplained gaps or duplicates |
| Consequential business actions | Loan-system events; Lineage Records; tool effects; applicant notifications | 14,903 | Reconcile every approve, decline, refer, credit pull, offer, and state change in both directions | 100% bidirectional match by `external_reference` or `effect_id`; amount and outcome agree |
| Control and exception events | Policy events; 3,842 Decision Requests; incident system; Assurance Alerts | 1,126 denials; 214 overrides; 367 anomaly or low-confidence events; 11 incidents or near misses | Reconcile each event to its action, owner, resolution, downstream effect, and evidence record | Every exception has a disposition; all affected actions remain in the audit universe |
| Shadow-agent search results | Model-gateway unknown clients; cloud principals; spend; browser extensions; automation vault | 3 candidates, 1 confirmed shadow agent | Trace each candidate to owner and use; add confirmed production activity to scope and population | All candidates resolved; confirmed shadow activity is contained, quantified, and reported |
3. Rank agents and action types by risk
Score the action type at its highest credible authority during the period. Use 0-3 for autonomy, reversibility, data sensitivity, financial or rights impact, tool authority, and evidence quality; use 0-2 for external dependencies and incident history. For evidence quality, 0 means independently verifiable and complete, while 3 means material gaps or mutable sources. The 22-point worked rubric uses Critical 17-22, High 12-16, Moderate 7-11, and Low 0-6.
The score orders fieldwork and sample coverage. It does not replace professional judgment. Raise the tier for credible catastrophic impact, active control bypass, an unresolved incident, or an incomplete population even when the arithmetic is lower. Document the evidence and approver for every manual adjustment.
| Agent action type | Autonomy | Reversibility | Data sensitivity | Financial / rights impact | Tool authority | External dependencies | Incident history | Evidence quality | Total / tier |
|---|---|---|---|---|---|---|---|---|---|
| Loan decline | 2 | 3 | 3 | 3 | 2 | 2 | 2 | 2 | 19 / Critical |
| Loan approval above EUR 25,000 | 1 | 3 | 3 | 3 | 3 | 2 | 1 | 2 | 18 / Critical |
| Cross-agent valuation delegation | 3 | 2 | 2 | 2 | 3 | 2 | 2 | 2 | 18 / Critical |
| Identity-document rejection | 2 | 2 | 3 | 3 | 2 | 2 | 1 | 1 | 16 / High |
| Third-party credit-data retrieval | 3 | 1 | 3 | 2 | 3 | 2 | 1 | 1 | 16 / High |
| Applicant status notification | 3 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 8 / Moderate |
| Document classification without a state change | 3 | 0 | 2 | 1 | 0 | 1 | 0 | 1 | 8 / Moderate |
4. Test control design against the 12 audit domains
A design test asks whether the stated control, if operated as written, addresses the identified risk. Inspect ownership, control objective, trigger, decision logic, authority, evidence capture, exception handling, frequency, retention, and escalation. Walk one normal case and one failure case through the design before testing period records.
The table reuses the enterprise framework taxonomy verbatim. It is the copyable audit test program for design. Add local workpaper references and results without changing the domain names, population, procedure, evidence, or pass criterion.
| Test ID and audit domain | Design population | Procedure | Evidence | Pass criterion |
|---|---|---|---|---|
| D-01: 1. Inventory and scope | 8 agents, 27 Releases, 31 Rollouts, 14 tools, 4 external dependencies | Inspect inventory schema and reconciliation control; trace one addition, one change, and one retirement through approval and production discovery | Agent Registry, Process map, inventory procedure, reconciliation output, approval record | Required objects, owners, environments, active dates, dependencies, and reconciliation sources are defined; unknown production activity triggers escalation |
| D-02: 2. Responsibility and accountability | 8 agents and all 12 control owners | Map outcome, risk acceptance, control operation, incident, evidence, and remediation decisions to named roles with authority and deputies | Responsibility matrix, role charters, committee mandate, escalation path | One accountable Process owner exists; every control and exception has an authorized owner and escalation route |
| D-03: 3. Identity and delegation | 19 persistent and 31 temporary identities; 286 delegated Journeys | Inspect unique identity, sponsor, purpose, scope, duration, token issue, expiry, and revocation design; walk a delegation and an emergency revoke | IAM standard, token claims, authority snapshot, delegation and revocation procedures | Each actor and delegation is unique, bounded, attributable, time-limited where applicable, and revocable before further use |
| D-04: 4. Permissions, tools, and data boundaries | 14 tools, 6 write-capable tools, 19 identities, 7 action types | Compare intended purpose to effective grants, Tool Catalog actions, resource scopes, Data Boundaries, value limits, and enforcement points | Access model, grants, Tool Catalog entries, Data Boundaries, deny tests | Least-privilege restrictions are enforced at action time; unregistered tools, broad wildcards, and direct bypass paths are blocked |
| D-05: 5. Risk classification and pre-production assurance | 27 Releases and 9 model or vendor changes | Inspect risk rubric, impact assessment, test plan, thresholds, exception approval, and release gate; trace one failed threshold | Risk assessments, model and Process evaluations, release criteria, sign-offs, accepted exceptions | Each Release maps material risks to tests and measurable thresholds; failures block release unless an authorized, bounded exception exists |
| D-06: 6. Runtime policy enforcement | 184,216 action records and applicable policy versions | Walk allow, deny, escalate, and exception paths; verify policy evaluates before execution and direct tool calls use the same gate | Policy Builder definition, KLA Policy Engine configuration, Simulation results, event sequence | Every consequential path requires an approved pre-execution verdict; denial and stale-policy states prevent the tool effect |
| D-07: 7. Human approval and escalation | 3,842 Decision Requests and 214 overrides | Inspect routing thresholds, reviewer authority, evidence presented, conflict handling, expiry, reassignment, rationale, and late-decision prevention | Decision Desk configuration, reviewer matrix, request schema, escalation timers | Required human decisions occur before execution by an authorized reviewer using defined evidence and a recorded rationale |
| D-08: 8. Execution lineage and business outcomes | 14,903 consequential actions | Trace intent, inputs, retrieved data, tool calls, policy, human decision, before and after state, downstream effect, and applicant notification | Lineage Record, Journey, tool receipt, loan-system record, notification reference | Stable identifiers reconstruct every material action and the recorded outcome agrees with the business system |
| D-09: 9. Continuous assurance and change management | 27 Releases, 31 Rollouts, 9 vendor or model changes, 367 alerts | Inspect change classification, regression trigger, approval, first-execution monitoring, alert thresholds, owner review, and remediation linkage | Release diffs, test plans, Rollout conditions, Assurance Alerts, review records | Every material change triggers reassessment and monitoring; failed signals create owned, time-bound remediation |
| D-10: 10. Incident response, revocation, and rollback | 11 incidents or near misses; 19 access revocations; 4 Rollbacks | Walk detection, triage, containment, affected-population query, credential revoke, Rollback, communication, recovery, and lessons learned | Incident plan, runbook, revoke and Rollback tests, severity and notification matrix | Owners can stop further action, identify affected cases, revoke authority, restore safe state, and preserve evidence within approved targets |
| D-11: 11. Evidence integrity, retention, and Independent Verification | 184,216 records in 91 daily evidence bundles | Inspect synchronous capture, manifest creation, hashes, signatures, custody, verifier access, retention, legal hold, and deletion controls | Evidence schema, manifests, key records, custody log, retention schedule, verifier result | A reviewer can reconcile, validate, and reperform sampled actions; alteration or missing records are detectable and escalated |
| D-12: 12. Multi-agent and third-party dependencies | 286 delegated Journeys, 4 external dependencies, 9 vendor or model changes | Inspect authenticated handoff, constraint propagation, version pinning, supplier evidence, incident duties, termination, and fallback | Handoff schema, gateway receipts, contracts, assurance reports, dependency inventory | Every contributing service and agent is attributable, authorized, versioned, constrained, and covered by evidence and incident terms |
5. Test operating effectiveness over the audit period
Operating-effectiveness testing uses actual period records. Run deterministic assertions across the full population where source completeness and field semantics are proven, then inspect sampled cases for judgment, context, and evidence quality. Preserve query text, source snapshots, row counts, exceptions, reviewer notes, and reperformance results in the workpapers.
Each exception remains tied to its denominator. A late policy verdict in 3 of 14,903 consequential actions has a different exposure from 3 cases in an unreconciled population. Quantify both the observed rate and the population limitation, then expand testing when the failure may be systemic.
| Test ID | Population and selection | Procedure | Evidence | Pass criterion |
|---|---|---|---|---|
| OE-01: Action records and policy sequence | All 184,216 records; separate 14,903 consequential actions | Test uniqueness, required fields, timestamp order, policy version, and pre-execution verdict; identify denied actions with a downstream `effect_id` | Frozen Lineage Record and policy-event extracts; query; exception file | 100% unique and attributable; every consequential action has an applicable prior allow or escalate result; zero denied actions execute |
| OE-02: Policy denials | All 1,126 denials analytically; 40 cases selected across rule, action type, agent, and Release for human review | Confirm requested action stopped, no equivalent retry bypassed the rule, notification and escalation followed policy, and evidence explains the decision | Policy result, matched rules, retry chain, tool gateway, notification, Decision Request | Zero prohibited effects or equivalent bypasses; every sampled denial is correct, timely, and reconstructable |
| OE-03: Human approvals | All 3,842 Decision Requests analytically; 60 sampled by risk, reviewer, outcome, and month | Test reviewer authority at decision time, evidence presented, decision timing, rationale, conflicts, expiry, and match to executed action | Decision Desk record, IAM snapshot, presented-evidence hash, tool receipt, business outcome | 100% required approvals precede action; sampled decisions have authorized reviewers, sufficient evidence, rationale, and matching effects |
| OE-04: Human overrides | All 214 overrides analytically; 50 sampled across original verdict, reviewer, reason code, outcome, and Release | Compare override authority and rationale to policy; trace original decision, new decision, tool effect, notification, and later incident or complaint | Original and override decisions, authority snapshot, rationale, Lineage Record, outcome | Every override is authorized, reasoned, timely, within delegated bounds, and accurately reflected downstream |
| OE-05: Business-state reconciliation | All 14,903 consequential actions analytically; 60 sampled for source inspection | Join requested action, approved action, tool response, before and after state hashes, amount, final loan status, and applicant notification | Tool receipt, loan-system event, notification, `before_state_hash`, `after_state_hash`, `external_reference` | 100% outcome and amount agreement; sampled source records support the recorded before and after state |
| OE-06: Releases and first executions | All 27 Releases; first and second consequential execution after each Release, 54 actions | Verify approval, artifact and policy versions, regression results, deployment time, expected monitoring, evidence completeness, and absence of pre-approval execution | Release manifest, test results, approval, Rollout record, first Lineage Records | Every Release is approved before use; all 54 first executions use the intended versions and pass control and evidence checks |
| OE-07: Access revocations | All 19 revocation events and all later activity for affected identities | Compare request, approval, effective time, token expiry, gateway enforcement, residual sessions, and subsequent denied or successful calls | IAM and token events, gateway logs, authority snapshots, incident links | Revocation meets the approved target; zero successful use occurs after the effective time |
| OE-08: Incidents and near misses | All 11 events and the full affected-action population for each | Reperform severity, affected-population query, containment, revocation, Rollback, notification, root cause, remediation, and closure approval | Incident file, Lineage Records, affected-case list, revoke and Rollback receipts, closure evidence | Every event has a complete affected population, timely containment, approved recovery, preserved evidence, and verified remediation |
| OE-09: Evidence integrity and retention | All 91 daily bundle manifests; 30 bundles selected for signature and chain validation; every sampled action | Recalculate manifest and record hashes, validate signatures and chain order, confirm retention class and legal hold, and resolve records through an independent verifier | Bundle manifests, signature keys, custody log, retention records, verifier output | All manifests reconcile to source counts; all 30 validations and every sampled record pass integrity, custody, and retention tests |
| OE-10: Vendor, model, and delegation changes | All 9 vendor or model changes and 286 delegated Journeys analytically; 50 Journeys sampled | Test change approval, contract and evidence terms, authenticated sender and recipient, propagated purpose and constraints, versions, sub-agent effects, and final outcome | Change record, contract, handoff receipts, authority snapshots, component versions, Journey outcome | Every change is approved before use; every sampled handoff is attributable, bounded, versioned, and traceable to the final outcome |
6. Select a reproducible risk and random sample
Freeze the population before selection. Record the source queries, extraction time, filters, random seed, hash method, sample owner, duplicate treatment, replacements, and final case list. Keep every applicable stratum tag on a selected case so one action can satisfy several coverage objectives without disappearing from exception reporting.
Full-population automated testing works for deterministic assertions over structured, reconciled records: uniqueness, required fields, timestamp order, policy-before-action sequence, amount matching, expired authority, denial followed by execution, and hash validation. Human review remains required for decision rationale, semantic consistency with purpose, relevance of retrieved data, quality of reviewer judgment, credible downstream harm, chain-of-custody exceptions, and opaque third-party evidence.
The selection counts below are KLA recommendations for this worked population. They are a defensible starting point for engagement planning. Document the expected assurance, tolerable deviation, population variability, reliance strategy, and expansion rules when the engagement needs statistical inference.
| Required stratum | Concrete population | Selection method and size | Human procedure | Evidence | Pass and expansion rule |
|---|---|---|---|---|---|
| High-risk action types | 2,412 Critical-tier declines, approvals above EUR 25,000, and cross-agent valuation actions | 60 cases: 20 highest-value or highest-impact, plus 40 spread across action type, agent, Release, and month | Reperform authority, policy, required human decision, tool effect, outcome, and evidence integrity end to end | Lineage Record, authority snapshot, policy, Decision Request, tool receipt, business state, evidence manifest | Zero unauthorized or unsupported consequential effects; any such failure expands to the full affected action type and Release |
| Random population sample | Base population of 184,216 action records; selection excludes cases already chosen for mandatory strata | 80 records selected by ascending SHA-256 of fixed seed plus `record_id`; retain the seed and query | Inspect end-to-end completeness and compare recorded purpose, action, and outcome with source records | All minimum evidence groups and authoritative source records | No unexplained missing or inconsistent field; one systemic schema gap expands to all records using that schema version |
| Exceptions and policy denials | 1,126 denials across 18 policy rules | Full-population automated sequence test; 40 human reviews covering every material rule and all 8 agents | Confirm the denied effect stopped, retry paths stayed governed, and escalation or notification matched policy | Policy decision, matched rules, retry chain, gateway record, Decision Request, notification | Zero denied effects; any bypass expands to all executions sharing rule, agent, Release, tool, or retry pattern |
| Human overrides | 214 overrides by 23 reviewers | Full-population authority and timing test; 50 human reviews including every reviewer with 5 or more overrides | Assess evidence presented, rationale, authority, conflict, timing, executed action, and later complaint or incident | Original decision, override, IAM snapshot, rationale, tool effect, outcome history | Every override is authorized, reasoned, prior to effect, and bounded; one invalid reviewer expands to all of that reviewer's decisions |
| Anomalous or low-confidence outcomes | 367 events from retry, outlier, drift, latency, data-quality, or low-confidence rules | 60 cases: 10 from each anomaly family, weighted to Critical and High actions | Validate the alert, investigation, disposition, decision route, effect, and remediation; inspect false-negative risk around thresholds | Assurance Alert, model and tool records, reviewer analysis, outcome, Remediation Plan | Every sampled alert is timely and correctly resolved; any missed required escalation expands to the family and threshold window |
| First executions after a Release | First and second consequential action after each of 27 Releases: 54 actions | Test all 54 | Match deployed versions, approval, regression results, policy, permissions, monitoring, outcome, and evidence schema | Release manifest, Rollout record, first Lineage Records, Assurance Alerts | All 54 use approved versions and pass controls; any failure expands to every action until correction or Rollback |
| Cross-agent delegation | 286 Journeys containing at least one agent-to-agent handoff | 50 Journeys across sender, recipient, tool, risk tier, and Release; include every three-hop Journey | Trace authenticated parties, delegated purpose, scope, expiry, constraint propagation, message integrity, tool effects, and final outcome | Handoff receipts, authority snapshots, component Lineage Records, tool receipts, Journey outcome | Every handoff is attributable and within delegated bounds; any broken chain expands to the sender-recipient-version combination |
| Incidents and near misses | 7 incidents and 4 near misses | Test all 11 and the complete affected-action population for each | Reperform detection, severity, containment, revoke, Rollback, notification, root cause, remediation, and closure | Incident file, affected-case query, action records, recovery receipts, closure evidence | Complete and accurate affected population with timely response and verified closure; any omission reopens the event |
| Vendor or model changes | 9 changes: 4 model versions, 2 retrieval vendors, 2 tool APIs, 1 orchestration service | Test all 9 changes and the first 3 consequential actions after each, 27 action reviews with overlap retained | Inspect due diligence, contract and evidence terms, risk reassessment, evaluation, approval, version pinning, monitoring, and fallback | Change approval, contract, evaluation, manifest, gateway records, first outcomes | Every change is approved and tested before use; any unapproved use expands to the complete exposure interval |
7. Evaluate evidence sufficiency for every sampled action
Evidence is sufficient when it is relevant to the assertion, reliable, complete, timely, traceable to the population, and protected against undetected change. Document the producer, authoritative system, extraction method, field semantics, retention, integrity mechanism, and chain of custody. Reperform the policy result and reconcile the downstream effect whenever the evidence supports those procedures.
Identity evidence must resolve the user or sponsoring principal, agent, service context, delegator, and approver. The shared schema uses `sponsoring_principal_id`, `agent_identity_id`, `authority_snapshot_id`, `delegated_by`, and `reviewer_id`; linked IAM records carry actor type, credential lifecycle, effective grants, and service context. Preserve the exemplar field names below so samples can move between the enterprise framework, Lineage Explorer, and Evidence Room without translation.
A screenshot, dashboard total, or management representation can corroborate a test and cannot replace the underlying period record. Where one required group is missing, classify the gap, identify the affected population, attempt an equivalent alternative procedure, and qualify the conclusion when the same assertion remains unsupported.
| Request ID and field group | Exact minimum fields | Requested source and format | Sufficiency and pass test |
|---|---|---|---|
| ER-01: Record and correlation | `record_id`, `occurred_at`, `environment`, `tenant_id`, `process_id`, `journey_id`, `correlation_id` | Immutable Lineage Record extract plus schema, timezone, query, row count, and extract hash | IDs are unique; timestamp order is coherent; record reconciles to the frozen population and complete Journey |
| ER-02: Agent and release | `agent_id`, `agent_release_id`, `model_id`, `model_version`, `orchestrator_version`, `prompt_template_version` | Agent Registry and signed Release manifest with immutable artifact references | Every version resolves to the artifact approved and deployed at `occurred_at` |
| ER-03: Principal and delegation | `agent_identity_id`, `sponsoring_principal_id`, `delegated_by`, `session_id`, `authority_snapshot_id`, `expires_at` | IAM, token, session, delegation, effective-grant, service-context, and revocation records | User or sponsor, agent, service context, and delegation are attributable, active, scoped, unexpired, and authorized for the action |
| ER-04: Purpose and risk | `purpose_code`, `decision_type`, `risk_tier`, `regulatory_classification`, `classification_basis_version` | Approved Process purpose, decision catalogue, risk assessment, and classification record | Recorded purpose and tier match the approved use and the classification effective for the Release |
| ER-05: Data provenance | `input_reference[]`, `source_hash[]`, `retrieval_query_hash`, `data_boundary_id`, `redaction_profile_id` | Protected inputs or resolvable references, retrieval receipts, source versions, Data Boundaries, and redaction record | Every material input and retrieved item resolves to the allowed source, time, boundary, and protected representation |
| ER-06: Tool action | `tool_id`, `tool_version`, `action`, `resource_scope`, `requested_args_hash`, `response_hash`, `effect_id` | Tool Catalog entry, gateway request and response, effective permission context, and effect receipt | Requested authority, actual call, response, and downstream effect agree and remain within approved scope |
| ER-07: Policy decision | `policy_id`, `policy_version`, `policy_decision`, `matched_rule_ids[]`, `exception_id`, `evaluated_at` | Policy Builder artifact, KLA Policy Engine event, matched rules, exception approval, and Simulation inputs | Auditor reperforms the same result; `evaluated_at` precedes the governed tool effect |
| ER-08: Human decision | `decision_request_id`, `reviewer_id`, `reviewer_role`, `presented_evidence_hash`, `decision`, `rationale`, `decided_at` | Decision Desk record plus reviewer IAM snapshot and the evidence presented at decision time | Approver identity and role are authorized; evidence, decision, rationale, and timing support the action taken |
| ER-09: Outcome and recovery | `outcome`, `before_state_hash`, `after_state_hash`, `external_reference`, `incident_id`, `rollback_id`, `revocation_event_id` | Authoritative before and after business records, notification, incident, Rollback, and revocation receipts | Recorded state and downstream effect match the business system; recovery events resolve and sequence correctly |
| ER-10: Integrity and retention | `evidence_hash`, `previous_record_hash`, `bundle_manifest_hash`, `signature_key_id`, `sealed_at`, `retention_class`, `legal_hold_id` | Sealed Evidence Bundle, manifest, signature and key record, custody log, retention schedule, and legal-hold record | Hashes recalculate; signature and chain validate; sealing is timely; custody, retention, and hold cover the sample |
8. Write findings and form a bounded audit opinion
Use a four-level severity model. Critical means an active or credible unauthorized, irreversible, safety, rights, or material financial exposure requiring immediate containment. High means a material or systemic control failure, significant affected population, or weak prevention over consequential actions. Moderate means a bounded control weakness with compensating controls and limited current exposure. Low means an isolated documentation or process weakness with low direct impact. Record likelihood, impact, velocity, detectability, population, compensating controls, and management risk acceptance behind the rating.
Every finding needs condition, criterion, cause, consequence, affected population, evidence, owner, remediation, and retest date. Separate a control exception from an evidence limitation. Where population completeness, evidence integrity, or a third-party dependency prevents the procedure, state the affected assertion and residual uncertainty in a qualified conclusion.
The final opinion names the scope, period, criteria, assurance level, domains tested, reliable populations, exceptions, and limitations. It reports whether controls were suitably designed and operated effectively for the stated engagement. It provides no certification of legal compliance, system safety, individual decision correctness outside the tested work, or future operation.
| Finding field | Completed finding: IA-AGT-2026-04 |
|---|---|
| Title and severity | Human overrides were executed without complete contemporaneous approval evidence: High |
| Condition | Across the complete population of 214 human overrides, 37 records had an empty `rationale` and 12 additional records used a `reviewer_role` that did not resolve to the approved underwriter matrix at `decided_at`. Five of those 12 decisions were recorded after the related write-capable tool effect. The 49 unique affected overrides represent 22.9% of the override population. |
| Criterion | Lending policy LND-04 v6.2 section 7.3 and Agent Control Standard ACS-02 v4.1 control HAO-4 require an authorized underwriter to review the presented evidence, record a decision and rationale, and complete the decision before any override reaches the loan system. |
| Cause | The June IAM migration omitted the contractor-underwriter group from the Decision Desk role mapping. The override API schema also allowed a blank rationale, and the tool gateway accepted an override reference without checking `decided_at` against the effect time. |
| Consequence | The organization cannot demonstrate contemporaneous, authorized judgment for 49 overrides. Five loan-system effects occurred before the recorded decision, creating a direct control bypass. Affected applicants may have received outcomes that cannot be supported from the preserved approval record, and management lacks reliable evidence for complaint, regulator, or internal review. |
| Affected population | 49 of 214 overrides between 1 April and 30 June 2026: 31 approvals, 13 declines, and 5 referral changes across 6 agents and 4 Releases. The five late decisions affected applications requesting EUR 286,000 in total. |
| Evidence | Full-population query IA-AGT-OE-04; frozen Decision Desk extract hash `8d61…c4a2`; IAM role snapshots; 49 Lineage Records; five tool receipts and loan-system events; policy LND-04 v6.2; control ACS-02 v4.1. Independent reperformance confirmed the counts and event order. |
| Owner and due date | Accountable owner: Head of Lending Operations. Supporting owners: IAM owner and agent technical owner. Remediation due 15 September 2026. |
| Remediation | Make `rationale` mandatory for every override; synchronize approved reviewer groups from IAM; require the gateway to validate reviewer authority and `decided_at` before a write; re-review all 49 affected cases; correct applicant and loan records where required; preserve the original gap and new review as separate evidence; monitor override completeness daily for 30 days. |
| Retest and closure criterion | Retest date: 30 September 2026. Reperform design tests for schema, role synchronization, and gateway sequencing; test the full 49-case remediation population and all overrides from 16-30 September. Close only when every affected case has an approved disposition, every new override has an authorized prior decision and rationale, and the 30-day monitor has zero failures. |
| Opinion effect | Qualify the operating-effectiveness conclusion for 7. Human approval and escalation and consequential overrides during the audit period. Preserve separate conclusions for other domains where sufficient evidence supports them. |
9. Report, follow up, and establish continuous assurance
Track each finding through one Remediation Plan with the owner, actions, due dates, dependencies, interim risk treatment, evidence, and status. Internal audit retests the failed assertion using the original criterion plus post-fix operating records. Closure requires verified design, effective operation for the defined observation window, complete treatment of the affected population, and approval under the audit methodology.
Run recurring analytics between audits over population reconciliation, policy denials followed by effects, late approvals, overrides, expired authority, Release and model changes, anomaly disposition, incident closure, evidence completeness, and signature validation. Management owns these controls and reviews. Internal audit validates the source, logic, thresholds, review evidence, and issue follow-through before relying on the analytics.
Trigger out-of-cycle audit work for a new high-impact agent, material Release, model or vendor replacement, permission expansion, new write-capable tool, Process or classification change, incident or near miss, unexplained population variance, evidence-integrity failure, repeated override or denial anomaly, or overdue Critical or High finding. Use the Agent Audit Readiness Assessment to score preparation, then use the 12-domain enterprise framework to expand any weak domain into deeper procedures.
- Remediation tracking: update owner, due date, action, dependency, interim control, evidence, and risk acceptance at least weekly for Critical and High findings.
- Control retesting: repeat the failed design step, test the complete known affected population, and inspect a defined post-fix operating window.
- Closure evidence: retain corrected configuration, approvals, affected-case dispositions, new period records, query results, verifier output, and audit closure sign-off.
- Recurring analytics: route breaches to an Assurance Alert with an owner, severity, due date, affected-population query, and Remediation Plan.
- Out-of-cycle work: open a scoped review when a trigger changes authority, impact, evidence reliability, or the audit population.
| Report section | Required content | Attached artifact | Completion criterion |
|---|---|---|---|
| 1. Executive summary and opinion | Assurance question, conclusion, severity profile, material exposure, reliable areas, and required action | Signed opinion and issue summary | Conclusion is bounded to criteria, scope, period, population, procedures, and evidence |
| 2. Objective, scope, and criteria | Business Process, agents, Releases, dates, decision types, materiality, exclusions, authority of each criterion | Approved engagement-planning checklist | Every test traces to the assurance question and an effective criterion |
| 3. System boundary and population | Architecture, owners, identities, tools, dependencies, counts, reconciliation results, and shadow-agent search | Population worksheet, source queries, hashes, and variance log | Population is complete or the exact limitation and affected assertions are stated |
| 4. Risk assessment and methodology | Risk rubric, ranked actions, design and operating-effectiveness approach, automated tests, sample method, seed, sizes, and expansion rules | Risk worksheet, test program, and sampling matrix | A reviewer can reproduce the selection and understand reliance on every procedure |
| 5. Results by audit domain | Design conclusion, operating conclusion, population tested, exceptions, rates, and evidence quality for all 12 domains | Workpaper index and evidence-request tracker | Every conclusion is supported by reviewed evidence and quantified exceptions |
| 6. Findings and management actions | Condition, criterion, cause, consequence, population, evidence, severity, owner, action, due date, and risk acceptance | Finding sheets and signed management responses | Actions address root cause and complete affected-population treatment |
| 7. Limitations and qualified conclusions | Missing evidence, failed alternatives, unavailable dependencies, residual uncertainty, and opinion effect | Scope-limitation and alternative-procedure workpapers | The reader can identify every assertion outside supported assurance |
| 8. Follow-up and Continuous Assurance | Remediation governance, retest plan, closure criteria, analytics, owners, thresholds, cadence, and out-of-cycle triggers | Remediation Plans, analytics inventory, and follow-up calendar | Every open issue and recurring signal has an owner, evidence source, due date, and escalation path |
| 9. Appendices | Definitions, criteria versions, detailed populations, queries, sample list, exclusions, evidence index, and severity rubric | Reproducibility package and Sealed Evidence Bundle references | An independent reviewer can trace report statements to workpapers without exposing protected data |
Häufig gestellte Fragen
What is an AI agent audit program?
It is the repeatable work plan for one assurance engagement: objective, criteria, complete population, risk assessment, control design and operating-effectiveness procedures, sampling, evidence tests, findings, opinion, and follow-up. The broader 12-domain enterprise framework defines the audit universe.
How should internal audit scope an AI agent engagement?
Start with the business Process and exact assurance question. Name the agents, Releases, identities, tools, models, dependencies, decision types, period, materiality, downstream effects, evidence stores, exclusions, and criteria. Reconcile the full population before selecting samples.
How should auditors sample AI agent actions?
Combine a reproducible random baseline with mandatory risk strata: high-risk actions, denials, overrides, anomalies, low-confidence outcomes, first executions after Releases, cross-agent delegation, incidents, near misses, and vendor or model changes. Preserve the population query, seed, selection logic, and expansion rules.
What evidence is required for one sampled AI agent action?
The record should resolve the principal and agent identities, delegation and service context, Agent and model Release, inputs and retrieved-data references, tool and permission context, policy result, human decision and rationale, before and after state, downstream effect, timestamps, retention, integrity proof, and chain of custody.
Can automated testing replace human sample review?
Automated tests can cover complete populations for deterministic assertions such as sequence, required fields, authority expiry, outcome reconciliation, and hash validation. Human review is still needed for rationale, context, semantic consistency, judgment quality, credible harm, and evidence limitations.
What should the audit report say when evidence is incomplete?
Classify the gap, quantify the affected population, attempt an equivalent alternative procedure, and state the unsupported assertion and residual uncertainty. Qualify the relevant domain or opinion boundary and assign remediation with a retest date.
Die wichtigsten Erkenntnisse
A usable AI agent audit program begins with a precise assurance question and a complete population. It ranks action risk, tests control design and actual operation, combines full-population analytics with reproducible samples, evaluates one consistent evidence schema, writes quantified findings, and verifies closure. Start with the Agent Audit Readiness Assessment, use the enterprise framework for deeper domain criteria, and compare the evidence request with the Evidence Room sample.