An invoice agent reads a supplier message, prepares a payment instruction and reports that the task succeeded. That sentence could describe several different tests. One checked the proposed instruction. Another called a simulated payment tool. A third inspected the resulting ledger entry and the approval that permitted it. Each test can produce a useful result, and each supports a different conclusion about the agent.
The scenario is illustrative. It frames one research question: which conclusion does a published agent evaluation support, given the system and the observations that produced it? This post reviews two recent preprints, an individual Internet-Draft and an older benchmark paper, reports a count of one published outcome file, and gives a boundary record that should travel with any headline score. This post is the second in Research notes on agent execution. The previous note covers how an AI agent’s success claim can influence its reviewer.
Write the intended claim before choosing a benchmark
Agent Evaluation Reliability: More Tasks Won’t (Always) Fix An Agent Leaderboard, a preprint submitted on 30 September 2026 by Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer and Sanmi Koyejo, separates two questions a leaderboard can answer. One asks which complete system ranks higher. The other asks which underlying model ranks higher. The authors apply a Bayesian variance decomposition to 22 benchmarks: nine from the Holistic Agent Leaderboard (HAL) and thirteen from the Harbor Index.
The authors report that rankings of fixed model and scaffold systems were more reliable than rankings of the underlying models. Their reliability measures describe how consistently a ranking holds for the sampled models, scaffolds and tasks. They say nothing about how often an agent completes a task, and they do not establish whether a ranking measures the right capability. Acceptance of the paper was not verified. Read the paper.
A scaffold is the software around the model: it manages state, exposes tools and turns model outputs into actions. A buyer selecting a complete agent application should include that software in the system being assessed. A team choosing a replacement model should treat the same software as a condition to hold fixed or to vary on purpose.
Write the intended conclusion in one sentence before building or reading an evaluation. For example: “This configuration can prepare an authorized payment instruction from these document types under this approval policy.” The sentence fixes the workflow, the action and the conditions the evidence must cover. A general statement about agent capability would need a much broader design.
Declare the tested boundary
Evaluation Profiles for AI Agent Security Assessment, by Han, Chen, Fan and Wu, was published on 23 October 2026. It organizes evaluation around infrastructure and scenario profiles, asks reports to declare the chosen profiles and their configuration, and asks for inapplicable metrics to be marked explicitly. The document is revision 00 of an individual Internet-Draft. It carries no IETF endorsement and has no standards standing; its use here is as a concrete reporting proposal under development. Read the proposal and its status.
For a business workflow, a boundary declaration names the parts of the system that produced the result and the party that judged it:
Persistence needs its own line. A clean session cannot show how an agent handles information carried across sessions, a question the note on policies in agent memory takes up. A simulated tool supports claims about the simulation. A claim about a live service needs evidence that covers the service interaction and the state it left behind.
- Model: snapshot identifier and inference settings.
- Scaffold: the operating software and its version.
- Tools and permissions: which tools the agent could call, and what each was allowed to change.
- Starting state: the fixture and how it was reset between cases.
- Effect destinations: the systems that receive effects, and whether each is simulated or live.
- Observer: what the grader could read. A grader reading the agent’s conversation holds different evidence from an observer reading an authoritative database after the run.
Check what the tools and the grader establish
Do Agent Benchmarks Do What They Say?, a preprint submitted on 29 September 2026 by Rohith Reddy Bellibatlu, Zichong Wang and Wenbin Zhang, treats each tool’s advertised behaviour as a contract and checks the implementation against it. Across four benchmarks, the authors report seven tool defects and one evaluator property at pinned commits. Known defects partly drove their choice of benchmarks, so the findings are examples and give no general defect rate. The authors’ earlier code snapshot uses different terms; we follow the paper’s classification. Read the audit.
The earlier MedAgentBench paper (version 2, 12 February 2025) shows a boundary stated in its own methods. Its evaluation setup sends GET requests to the environment, checks POST payloads and signals execution success to the agent; action grading checks the requested payload. The authors document this design, so readers can interpret the action results within it. Read the original methods, sections 2.4–2.4.3.
For an invoice workflow, keep four stages apart and preserve each one that exists. If the evaluation stops at one of them, make that stopping point visible in the result.
- Proposed instruction: what the agent intended to do.
- Tool request: what the agent actually sent.
- Tool response: what the tool reported back. A success message is the tool’s own claim.
- Recorded effect: what an independent observer found in the system of record afterwards.
A worked boundary record for a supplier payment
Consider a fictional supplier-payment workflow with three evaluation designs. The input is fixed: supplier SUP-042, invoice INV-042 for EUR 1,200, and an instruction to prepare one payment that a finance operator must approve before any payment is recorded. Payment release exists only inside a local simulation. No model, simulator or payment workflow was run, and the model snapshot, harness and evaluator versions are still unselected. The table is an extract of the full evaluation-boundary record.
The supplier, amount and required approval are identical across the three designs, and the result columns still stay separate. Adding persistent memory introduces new behaviour to observe. Moving the tool host moves the place where permissions and effects are checked. Design C also runs two sessions, so its resource budget differs and the record reports that difference. Identical model snapshots alone would not make the three scores interchangeable.
| Field | A: instruction preparation | B: approval simulation | C: persistent workflow simulation |
|---|---|---|---|
| Tools | None | read_invoice, request_approval, record_payment in a local simulation | B plus write_supplier_memory and read_supplier_memory |
| Permission boundary | Output generation only | Simulator checks actor, approval identity, invoice and amount | B plus supplier and session memory access |
| Evaluator access | Input fixture and output | Fixture, full tool trace, decision record, simulator state before and after | B plus memory snapshots and the read and write trace |
| Consequential effect | No payment operation exists | Local payment-state transition | Local payment-state transition following persisted reads |
| Exclusions | Execution, approval enforcement, memory, network effects | Real payment rail, cross-session memory, delegation | Real payment rail, cross-user isolation, direct storage tampering, delegation |
| Claim within reach after execution | Correct preparation for the supplied cases | Correct behaviour within the declared simulation | Correct behaviour across the tested session sequence |
| Execution status | Unrun | Unrun | Unrun |
Give every case a status, including cases that did not run
The record defines three cases: an ordinary authorized payment, an instruction with no approval, and a second session that reuses a supplier record stored in the first. They show the intended coverage. They do not estimate how often anything fails in practice, and selecting them shows nothing about whether an implementation passes. The second-session case does not test corruption, revocation or deletion of the stored record, and the missing-approval case gives no estimate of adversarial robustness.
Each result carries one of five labels. The evaluation-profile draft asks for inapplicable metrics to be marked; the rest of this vocabulary is our own proposal, and no group has adopted it. In this example every execution-dependent result is Unrun, and the record defines no aggregate score.
- Publish per-case evidence availability and the denominator behind any rate.
- Keep missing observations and inapplicable controls out of the success count.
| Label | Use |
|---|---|
| Pass or fail | A defined criterion evaluated with sufficient recorded observations |
| Not applicable | The capability is absent from the declared configuration |
| Not exercised | The capability exists, and the selected case never reached it |
| Unobserved | The observation needed for a verdict is unavailable |
| Unrun | Execution has not occurred |
What we checked in the published data
The reliability paper’s authors publish their outcome data. We downloaded the HAL outcome file from their repository at commit e7ec464 (29 September 2026) and counted it with the Python standard library: 29,923 rows, 9 benchmarks, 54 model-name values, 13 scaffold names and 1,117 distinct benchmark and task pairs, with every score either 0 or 1. These dimensions match the repository’s own data description. The model names include reasoning-effort variants, so 54 is a count of identifiers and overstates the number of base models. Pinned repository, pinned file.
This was a data count. We did not rerun any agent, inspect trajectories or refit the statistical model, and the count cannot show that the labels and grades in the file are correct. It confirms the shape and identity of the input that the reported reliability figures rest on. The reliability figures themselves remain the authors’ results.
Publish the boundary record with the result
Attach the boundary record to a result under the same version identifier, and fill in the run fields when the evaluation executes:
When a component changes, list the change beside the new result so readers can judge what a comparison means. A reader can then trace a conclusion back to a particular configuration, action and observation, and an assurance team can decide what further evidence its own workflow needs.
- Run ID and exact versions: model snapshot, harness, tool host, evaluator and dataset hash.
- Sampling settings or seed, the resource budget and what the run actually consumed.
- Reset evidence, actor permissions and the identity of any human who decided.
- Observer access, initialization failures and exclusions, with how each enters the denominator.
- Action trace and state differences, with tool success messages kept apart from independently observed effects.
- The verdict rule applied to each case.
Frequently Asked Questions
What does an AI agent benchmark score measure?
A score measures one configuration on one set of tasks under one grader: the model and its settings, the scaffold, the tools and permissions, the starting state and what the grader could observe. Whether it says anything about the model alone, or about a live service, depends on how those parts were held fixed or varied.
Should you evaluate the AI model or the whole agent system?
Follow the claim you need. A buyer choosing a complete agent application should evaluate the model together with its scaffold. A team choosing a replacement model should hold the scaffold fixed or vary it on purpose. A September 2026 preprint by Hardy and colleagues reports that complete systems ranked more reliably than underlying models in the benchmarks it studied.
What should an AI agent evaluation report include?
The intended claim, exact versions of the model, scaffold, tools and evaluator, permissions, reset procedure, resource budget, observer access, exclusions and initialization failures with their effect on the denominator, and a status for every case: pass, fail, not applicable, not exercised, unobserved or unrun.
Is there a standard for reporting AI agent security evaluations?
The reporting proposal reviewed in this post has no standards standing. Evaluation Profiles for AI Agent Security Assessment (23 October 2026) proposes declaring infrastructure and scenario profiles and marking inapplicable metrics. It is revision 00 of an individual Internet-Draft, with no IETF endorsement. Its companion methodology document could not be retrieved on 26 October 2026.
Key Takeaways
A benchmark score supports the conclusion its configuration and observations can carry. Write the intended claim, declare the boundary, keep tool responses apart from recorded effects, and give every case a status. Start with the evaluation-boundary record, and read the AI agent audit program for how assurance teams scope and sample the evidence behind agent actions.
Sources were checked on 26 October 2026. Both papers are preprints, and their results remain the authors’. The evaluation-profile document is an individual Internet-Draft with no standards standing. The worked record is synthetic and unrun; no agent experiment was run for this post. Our data check counted one published file and reproduced no statistical result. This post contains no KLA result.