Glossary
/
AI Agents

Evals

Structured tests that measure what an AI agent gets right and wrong.

Definition

Evals (evaluations) are structured tests that measure how often an AI agent gets the right answer. A team runs the agent against a set of real questions, scores each answer and uses the mistakes to improve it. Evals decide when an agent is ready to go live.

Key points

<ul><li>Example: test an agent on 200 real customer questions and require it to meet an agreed accuracy threshold before launch.</li><li>Evals keep running after launch, so the team can catch new mistakes and use them to improve the agent.</li><li>Evals are also a fair way to compare vendors: same questions, compared on accuracy, speed and cost.</li></ul>