Explainability

Explainability is the degree to which people can understand how an AI system reached an output or decision and explain it to others. It has two sides: technical interpretability, which concerns how the model works internally, and decision-level explanation, which tells a reviewer or an affected person why a particular case came out the way it did.

Three provisions of the EU AI Act depend on it. Article 13 requires high-risk AI systems to be "sufficiently transparent to enable deployers to interpret a system's output and use it appropriately", with instructions for use that describe the system's capabilities and limitations. Article 14(4)(c) requires the people assigned to human oversight to be able to interpret the system's output correctly. Article 86 gives a person affected by a deployer's decision, taken on the basis of the output of a high-risk AI system listed in Annex III and producing legal or similarly significant effects, the right to obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken."

The explanation has to suit its audience. A model validator may use feature-attribution methods such as SHAP or LIME. A reviewer approving an agent's action needs the facts the decision rested on, the rule or threshold that applied, and what the agent proposed to do. An affected customer needs the main reasons in plain language.

For agents built on large language models, the model's own account of its reasoning can be wrong. Decision-level explanations are more reliable when they come from records: the inputs the agent read, the policy version and rule that produced the outcome, and the human decision where one was needed. Document the approach in the Annex IV technical documentation: the methods used, what is captured for each decision, and who can see it.

In KLA, each policy outcome is recorded with the rule that produced it, and the Lineage Explorer shows the tool calls and inputs behind a run. In our own QA rounds with test agents, outcomes were easiest to review when each one carried the rule's stated reason and the exact policy version and hash.