Production AI Agents Need Policies, Evals, and Escalation

A model can produce a convincing support or sales conversation in a demo. Production begins when the system must follow company policy, access only the right data, take an approved action, recover from uncertainty, and involve a person before the cost of a mistake becomes unacceptable.

Start with one job and one owner

Define the request the agent handles, the knowledge it may use, the actions it may take, and the person accountable for the outcome. A narrow job creates a testable contract. A general promise to help with anything creates an evaluation problem with no meaningful boundary.

• Represent operating policy separately from conversational style
• Allow only the tools and data required for the defined job
• Require approval for high-impact, unusual, or irreversible actions
• Escalate with the transcript, gathered context, and attempted steps intact
• Review production failures and add them to the permanent evaluation set

Evaluate the decision, not just the wording

A fluent answer can still choose the wrong policy, call the wrong tool, or escalate too late. Measure whether the agent retrieved the correct evidence, respected permissions, completed the approved action, and handed off when it lacked authority. Language quality matters, but operational correctness decides whether the system is safe to scale.

Improve through controlled releases

Use production sessions and escalations to propose policy, prompt, retrieval, and tool changes. Test each change against the same representative cases, compare it with the current version, and roll it out deliberately. Dependable agents improve continuously while the business remains in control.