Evaluating Long-Horizon Agents Beyond the Final Answer

A long-horizon agent can reach the correct final state through a dangerous sequence, or fail after doing several useful steps. A single pass/fail label hides both realities. Evaluation needs to inspect the trajectory as well as the destination.

Score the workflow in layers

Separate task understanding, plan quality, tool selection, argument validity, state tracking, recovery, final verification, and user communication. This shows whether a regression came from the model, the tool layer, context assembly, or product policy.

• Completion: did the requested outcome actually occur?
• Safety: were permissions, approvals, and data boundaries respected?
• Efficiency: how many turns, tokens, retries, and external calls were used?
• Recovery: did the agent detect and correct tool or environment failures?
• Evidence: can a reviewer verify the final claim from recorded artifacts?

Build representative scenarios

Include ordinary work, edge cases, ambiguous requests, missing access, stale data, and adversarial content. The set should reflect the actual workflow distribution, not only examples that the current system already handles.

Keep humans in the evaluation loop

Automated graders scale coverage, while domain reviewers catch subtle errors and shifting business expectations. Use both, review disagreements, and promote newly discovered failures into permanent regression cases.