Model Harnesses, Routing, and Substitutability

TL;DR

• An AI product is the model plus the harness that supplies context, tools, permissions, state, an execution environment, and feedback.
• Start with one model behind a clean application boundary. Add routing only when task-level evidence shows a material gain in quality, latency, resilience, or cost.
• The durable asset is the workload-specific harness: authoritative context, tool contracts, traces, evals, and operating evidence.

---

Model rankings move quickly. Broad benchmarks compress many capabilities into one score and rarely reflect your inputs, tools, latency target, safety requirements, or cost profile.

Production performance depends on more than the model. Missing context, ambiguous tools, excessive permissions, weak recovery, or a poor feedback loop can make a capable model fail repeatedly. Changing providers will not repair a broken surrounding system.

Treat the harness as the product architecture. Routing is one component it may contain.

The harness is the product surface

The harness is everything that turns a general model into a product capable of doing a specific job.

A weak harness asks the model to compensate for missing product decisions. The prompt grows, permissions widen, and retries multiply. A strong harness gives the model enough verified information and bounded capability to complete the job, then makes failure visible when it cannot.

Context is an operated dependency

Context quality often matters more than context volume. More documents can introduce stale instructions, conflicting definitions, irrelevant examples, and permission leaks.

Build context assembly around four rules:

1. Retrieve from an authoritative source rather than duplicating mutable policy into prompts.
2. Select context for the current task instead of loading everything the system can access.
3. Preserve provenance so users and operators can inspect what influenced the result.
4. Test context changes against the affected eval slices before release.

Memory needs the same discipline. Separate durable user preferences from task state, organisational knowledge, and interaction history. Give each class a retention rule, correction path, and permission boundary. A plausible memory that cannot be traced or corrected becomes a product liability.

Every Agent Needs an Owner covers context ownership, expiry, and lifecycle responsibilities. The security boundaries chapter covers untrusted context and memory poisoning.

Preserve substitutability without abstraction theatre

No model is best for every workload forever. That does not mean every product needs a universal provider abstraction or a router on day one.

Start with the simplest model setup that clears the quality, latency, privacy, reliability, and cost bar. Keep a clean boundary between application logic and the provider client. Store representative eval cases. Document the provider-specific features the product depends on, including tool schemas, caching, structured output, context behaviour, and safety controls.

This creates a credible migration path without pretending models are interchangeable. Prompts, tools, and failure modes still need revalidation when a provider or model changes.

Add another provider, model, or abstraction layer when the measured switching risk justifies its ongoing cost. Portability is valuable. Premature portability is another form of architecture theatre.

Routing is an earned optimisation

A routing layer inspects a request or task state and selects a model path. The decision may account for a hard capability requirement, quality floor, latency target, availability, data constraint, or cost ceiling.

Classification can be rule-based, model-based, or hybrid. The simplest reliable rule should win. A product that already knows the task type from the workflow should not pay another model to rediscover it.

A useful routing decision follows this sequence:

1. Apply hard requirements such as modality, residency, context size, or approved-provider policy.
2. Remove candidates that miss the measured quality or latency floor for the task.
3. Choose the lowest-cost qualifying path, including expected retries and review.
4. Define failover and recovery for provider errors or quality-check failures.

The router needs three operated assets:

Task taxonomy. A classification of the work the product actually receives. Derive it from production evidence rather than a vendor capability list.

Quality baselines by task. Representative eval results showing which paths meet the required standard. Re-run the relevant comparison when models, prompts, tools, or task distributions change.

End-to-end operating profiles. Measured latency, cost, retry, and failure data for the complete path. Provider prices and benchmark latency are inputs, not production evidence.

Routing becomes useful intellectual property when these assets improve with real work. The switch statement itself is easy to copy.

Worker-manager systems pay an audit tax

Some workflows split a job between a manager and several workers. The manager plans, delegates, aggregates, or reviews. Workers perform bounded tasks such as extraction, search, formatting, or validation.

Use this pattern when decomposition creates a measured benefit, not because the org chart looks sophisticated. A single capable agent with the right tools is usually easier to evaluate and operate.

Oversight can dominate the economics when an expensive manager reviews every worker output. Model the full trace with the formula in Business Viability, including retries, duplicated context, review tokens, tool calls, and human exceptions.

Reduce the audit burden through controls supported by evidence:

• Deterministic validation for structured or rule-bound work
• Risk signals calibrated against observed outcomes
• Statistical sampling of apparently successful work
• Full review for new, changed, or consequential task types
• Automatic escalation when quality, cost, or loop limits are breached

Do not route work based only on a model's self-reported confidence. The review policy should combine failure consequence with signals validated against real outcomes.

Tools need stable contracts

Tools turn a model response into action. Their descriptions, schemas, permissions, error semantics, and versioning are product interfaces.

Model Context Protocol (MCP) can decouple compatible tools from a specific model client. It does not remove the operating work. Every exposed server adds attack surface, latency, permissions, and a contract that can change.

For each tool or MCP server, decide:

• Which job requires it
• Which data and actions it exposes
• How parameters are validated outside the model
• What happens on timeout, partial success, or duplicate execution
• How breaking changes are versioned and tested
• Which traces and denied actions operators can inspect

Clear tool descriptions help the model select and use a capability. Deterministic validation controls whether the requested action is permitted. Do not ask the model to enforce its own authority boundary.

Engineer cost and latency across the trace

The visible generation is only one part of the operating cost. Include context retrieval, cached and uncached input, tool calls, retries, safety checks, orchestration, human review, and failure recovery.

Three product decisions often matter more than elaborate routing:

Context structure and caching. Stable repeated context may qualify for provider-specific caching. Optimise first for clarity and eval performance, then measure the saving on the real workload.

Synchronous versus asynchronous work. Research, batch classification, and scheduled jobs may not need an interactive latency target. Background execution can improve economics if progress, cancellation, recovery, and notification are designed properly.

Retry and escalation policy. Blind retries can multiply cost while reproducing the same failure. Retry only when the failure is plausibly transient or a different validated path can help. Escalate or abstain when required information is missing.

When routing earns its complexity

Route across models when:

• The product handles materially different task types with measured differences between candidate models
• Inference spend is large enough for allocation savings to exceed engineering and operating cost
• A single-provider outage would breach an important availability commitment
• Data, deployment, or capability requirements force distinct paths
• The team can evaluate quality and economics by route

Stay with one model when:

• The product performs one focused job
• Volume or spend is low
• The team cannot measure quality by task type
• Provider differences are smaller than the maintenance burden
• Most failures trace to context, tools, permissions, or workflow design

Calculate the break-even point. Include engineering time, incident load, duplicated prompts, evaluation cost, and commercial commitments. If the benefit is not clear, keep the architecture simple.

What harness-aware PMs look like

The anti-pattern: provider entanglement

The team couples product logic, tool schemas, output parsing, and evaluation to one provider's current behaviour. Nobody records the dependencies because the system works.

A model is deprecated, pricing changes, or the primary task regresses. Switching now requires a rewrite and the team lacks representative evals to prove that the replacement is safe.

The failure was not using one model. Starting with one model was reasonable. The failure was allowing a provider implementation to become the product architecture without a tested boundary, dependency record, or migration evidence.