Evaluation Frameworks as Product Infrastructure

TL;DR

• Evals are day-one infrastructure, not post-launch monitoring. If you can't measure quality, you can't iterate.
• Start with a small representative set drawn from real tasks and known failures. Grow it into capability benchmarks and regression coverage as evidence accumulates.
• For agents, grade the path and the destination. Trace grading catches loops, wasted tool calls, and cost blow-outs even when the final output is correct.
• Give the quality definition an accountable owner. Expert disagreement should improve the rubric and test set, not disappear inside an average score.

---

Most AI teams treat evaluation as a quality phase that happens after the product works. Write the prompts, build the tooling, test it manually, ship it, then maybe add some evals when things start breaking.

That sequence leaves the team without a stable definition of good while the system changes fastest. Evals make those changes comparable. They show which tasks improved, which regressed, and whether the result still clears the release threshold.

An eval suite is more than a test harness. It expresses the product's acceptance criteria and gives product, engineering, domain experts, and leadership a shared quality language. Passing evals is necessary for release, but operational readiness also includes security, economics, recovery, and human workload.

Why evals come first

Traditional software has a well-understood testing story. Unit tests, integration tests, end-to-end tests, CI/CD pipelines. AI products inherit none of this for free.

A model can be technically healthy (low latency, no errors, serving responses) while producing increasingly wrong answers. A prompt change that improves one capability can silently degrade four others. A model upgrade that scores better on public benchmarks can perform worse on your specific use cases.

You cannot observe these problems through logs and dashboards. You need structured evaluation that runs automatically, compares against known-good baselines, and surfaces regressions before they reach users.

Three reasons to define evals early:

The most volatile phase is the least measured. Early development produces the largest behavioural swings. Each prompt iteration, each tool change, each model swap shifts output dramatically. This is exactly when measurement matters most, and exactly when most teams rely on manual spot checks.

Measurement shapes the work. Teams that build evals early write more precise requirements. Instead of "the agent should handle refunds well," they define valid outcomes, permitted tools, escalation conditions, latency, and the quality threshold required by the consequence of failure.

Retroactive suites lose evidence. Building evals after launch means reconstructing cases from memory and incomplete incident records. Capture representative examples and failures while their context, expected result, and consequence are still available.

Building the eval suite

The growth path for an eval suite has three stages, each building on the last.

Stage 1: Seed examples

Your first eval examples come from three places:

Production failures. Bug reports, support tickets, and expert corrections can become test cases when the team preserves the input, relevant context, expected behaviour, and reason the result matters.

Pre-release rituals. Every team has things they manually verify before shipping. "Does it still handle cancellations?" "Does it pick the right tool for address lookups?" Script these checks. Automate the things you're already doing by hand.

Edge cases from domain experts. Subject-matter experts know the ambiguous requests, rare but consequential inputs, and scenarios where the system should refuse rather than guess. Capture enough to represent the important boundaries rather than chasing an arbitrary count.

A small, well-chosen set is more useful than a large undifferentiated one. Synthetic data can expand coverage, but it reflects the generator and rubric used to create it. Combine it with real tasks, production failures, domain expertise, and important segments.

Stage 2: Capability benchmarks

As the seed suite stabilises, add capability evals that target what the agent struggles with. These start with low pass rates and give your team a hill to climb.

If an agent performs well on straightforward queries and poorly on complex multi-step tasks, create a capability slice for those tasks. This makes progress and trade-offs visible instead of hiding them inside an aggregate score.

Capability evals use human-graded gold standards. A domain expert reviews the expected outputs and confirms they represent genuinely correct behaviour. Without this calibration step, you risk optimising toward outputs that pass automated checks but fail in practice.

Own the rubric and disagreements

Evaluation work can stall when a committee tries to agree on every label before the team learns anything. It can also become arbitrary when one person's preference is treated as ground truth without inspection.

Assign an accountable quality owner for each important task family. This person maintains the rubric, resolves routine ambiguity, and ensures the eval remains useful for product decisions. They should have enough domain depth to recognise a plausible but wrong result and enough product context to understand the consequence.

Use other experts to calibrate the owner and expose disagreement:

1. Have two or more reviewers independently grade a representative sample.
2. Inspect cases where they disagree.
3. Determine whether the rubric is unclear, the source evidence conflicts, or the task has several acceptable outcomes.
4. Update the rubric, examples, abstention rule, or product boundary.
5. Re-grade the affected slice before using it as a release gate.

Inter-rater agreement is a diagnostic, not the outcome. High agreement on a weak rubric produces consistent error. Low agreement may reveal a product requirement the team has never made explicit.

Keep comments and reasoning for consequential disagreements. The decision history becomes part of the quality definition and helps new reviewers understand why the boundary exists.

Stage 3: Regression suite

As capability evals improve and pass rates cross your quality threshold, those tasks graduate into the regression suite. What once measured "can we do this at all?" now measures "can we still do this reliably?"

Regression evals protect known-good behaviour. Run the relevant slices on changes that can affect them, such as prompt edits, model swaps, tool modifications, and configuration changes. Reserve the full suite for the cadence and risk level that justify its cost. Any breach of a release floor should stop promotion while the team investigates.

The lifecycle is natural: seed examples catch the failures that already hurt you, capability evals push toward better performance, and regression evals lock in those gains.

Grading path and destination

For a chatbot, the final answer is what matters. For an agent, the execution trace is just as important.

Consider a support agent that successfully processes a refund (the agentic AI patterns chapter covers these workflow architectures). The outcome is correct. But the trace reveals that the agent called the identity verification tool three times unnecessarily, re-read a policy document it had already retrieved, used an expensive reasoning model to format a confirmation email, and took 15 turns to complete a 5-turn task.

The outcome passed. The execution was terrible. In production, that waste translates directly into inference cost, latency, and user frustration.

Outcome grading

Checks whether the end state is correct. Did the refund process? Did the code compile? Did the report contain the required sections? Binary, verifiable, essential.

Outcome graders come in three flavours:

• Deterministic graders check pattern matches, schema conformance, compilation, format validation. Fast, cheap, and objective.
• Model-based graders use a language model to judge output against a rubric ("Was the explanation grounded in retrieved documents?"). More expensive and non-deterministic, but they capture nuance.
• Human graders are the gold standard for calibration and genuinely novel scenarios. Expensive and slow, so use them strategically.

Trace grading

Checks how the agent got there. Trace grading evaluates the full execution path against efficiency and correctness criteria:

• Turn count. Did the agent complete the task in a reasonable number of steps?
• Tool selection. Did it use the right tools? Did it call expensive tools when cheap ones would suffice?
• Loop detection. Did it revisit the same tool or re-read the same document unnecessarily?
• Backtracking. Did it abandon a correct approach and try an inferior one?
• Escalation timing. When confidence was low, did it escalate promptly or muddle through?

Both dimensions matter. A correct outcome achieved wastefully is a margin problem. An efficient trace that produces the wrong outcome is a quality problem. Measure both.

Evaluate adversarial behaviour adaptively

Security and misuse evals need more than a fixed set of attack prompts. Static regression cases are useful after a failure is known, but an attacker can observe the system and change the input.

Build task-specific adversarial slices around the production boundary:

• Attempts to override the intended job
• Malicious instructions inside retrieved or uploaded content
• Tool calls that exceed the user's intent or permission
• Cross-tenant and sensitive-data access
• Encoded, fragmented, multilingual, and multimodal attacks
• Combinations of individually permitted steps that produce a prohibited outcome
• Failure of approval, rollback, timeout, and shutdown controls

Use adaptive human testing where consequence warrants it. Testers should change their approach after seeing the defence rather than replaying a frozen dataset. Record successful paths in the regression suite while continuing to search for new ones.

The agentic security playbook defines the authority and containment controls these evals should exercise.

Eval dimensions beyond accuracy

Accuracy is necessary but insufficient. A production eval suite measures across multiple dimensions.

Track these dimensions per task type, not just in aggregate. An agent that's fast and cheap on simple queries but slow and expensive on complex ones needs different optimisation than one that's uniformly mediocre. The AI product metrics chapter covers the product measurement layer that sits above these eval dimensions: adoption, trust calibration, and value delivery speed.

Continuous system calibration

Evals protect known behaviour before deployment. Continuous calibration finds what changed after release across the model, context, tools, users, and surrounding workflow.

Production creates patterns the original eval set could not contain. Users attempt adjacent jobs. Policies change. A tool returns a new error shape. A successful feature attracts a different segment. Monitoring only the model misses most of this movement.

Three types of drift demand automated monitoring:

Data drift. The statistical properties of incoming inputs are changing. Users ask about topics, in formats, or at volumes the system wasn't built for. A property valuation model trained on suburban housing data starts receiving commercial property queries. The inputs look different. The model doesn't know it's out of distribution.

Prediction drift. The distribution of model outputs shifts even though inputs look similar. A classification model that used to distribute predictions across five categories starts concentrating on two. Something changed in the model's behaviour, and the cause isn't obvious from inputs alone.

Concept drift. The real-world meaning of the data has changed. What counted as a "high priority" support ticket six months ago has different criteria today. The model may be unchanged while its accuracy degrades against a changing reality.

Drift metrics catch movement that the team already knows how to define. Pair them with sampled trace review, correction analysis, support evidence, user research, and incident review to find new failure classes.

Turn signals into the right response

Not every production defect needs another grader. The goal is better product behaviour, not the largest eval suite.

Alerting and response

Automate monitoring where traffic, change rate, and consequence justify it. When a drift signal crosses a validated threshold, the system should:

1. Alert the product team
2. Create an investigation item with the affected segment and evidence
3. Log the drift event with enough context to diagnose the cause

A frozen model does not guarantee a stable system. Inputs, user behaviour, policies, tools, context, and real-world concepts can move independently. The response may be a source-data repair, retrieval change, prompt or model update, revised threshold, narrower product boundary, or retraining. In regulated environments, connect monitoring and change control to the AI governance framework and current specialist advice.

The HITL feedback loop

Human-in-the-loop (HITL) workflows serve two purposes: quality control and data generation. Most teams understand the first. The second is where the real value sits.

The process

1. Route. Send work to a human when a validated risk signal, policy rule, or sampling plan requires review. Do not use an uncalibrated confidence score as the only control.
2. Refine. The human corrects, validates, or improves the output.
3. Feed back. The corrected output becomes a new labelled example, ground truth that feeds directly into your eval suite and, optionally, into fine-tuning datasets.

Why this matters for evals

Every human correction generates a high-quality test case. The input is real production data. The expected output is expert-validated. Over time, the HITL workflow becomes a data generation engine that continuously expands and strengthens your eval suite.

This can create a useful loop: the eval suite identifies weak spots, human review produces corrections, and suitable corrections become new eval cases. Quality improves only when the review is accurate, representative, and fed back into product decisions.

Teams that operationalise this loop build a stronger decision asset. Their eval coverage reflects real customers and failures, making future model, prompt, and workflow choices easier to compare. Usage alone creates no advantage unless the evidence is captured, governed, and used.

Evals as decision infrastructure

When a new model becomes available, teams without evals must reconstruct how to compare it. Is it safe for the use case? Does it improve the important tasks? Which segments regress?

Teams with representative eval suites can run the relevant comparisons and inspect the trade-offs. They may adopt an improvement sooner, reject a regression, or adjust multi-model orchestration with evidence. The durable asset is the workload-specific quality definition, not allegiance to one model generation.

The advantage extends beyond model adoption:

Faster prompt iteration. Every prompt change runs against the full suite before merging. Engineers experiment freely because the evals catch regressions automatically.

Clearer product decisions. Instead of debating in meetings whether the agent is "good enough," there are numbers. Before-and-after scores replace opinions.

Stronger vendor negotiation. Workload-specific quality, latency, and cost evidence makes provider alternatives credible during commercial discussions.

Faster onboarding. New engineers understand the product's quality bar immediately by reading the eval suite. The evals document what "correct behaviour" means more precisely than any spec.

What eval-driven PMs look like

The anti-pattern: the vibes-based launch

The team has been building an agent for three months. It works well in demos. The founder tested it last week and liked the results. A few engineers have run it through their favourite test cases manually. Everyone feels good about it.

They ship.

Within a week, support tickets surface failures nobody anticipated. A prompt fix for one issue breaks two others. The team can't tell which changes helped and which hurt, because there's no baseline to compare against. They enter a reactive cycle: patch, deploy, hope, repeat.

Six weeks later, a new model version drops. The team wants to adopt it but can't justify the risk. Manual testing would take two weeks. So they stay on the old model, falling behind competitors who upgraded in a day.

The lasting cost of a vibes-based launch is the inability to compare changes. Without a baseline, the team cannot tell whether a fix improved one task while degrading another. Evals turn those arguments into inspectable evidence.

Start with a representative seed set, grow it into capability and regression slices, and add production monitoring where the risk and traffic justify it. The agent safety inspection extends that evidence into operational boundaries and recovery.