Production traffic becomes the evaluation source
Microsoft has published a workflow for improving production agents by connecting live traces to continuous evaluation and controlled optimisation. The central idea is straightforward: real users expose failure modes that pre-release test sets miss. Foundry can turn selected traces into datasets, score them and feed the evidence into Agent Optimizer rather than relying on anecdotal prompt edits.
This matters because an agent can degrade without a code deployment. User behaviour changes, connected data evolves and a new model version may trade accuracy for speed or cost. A closed loop gives teams a repeatable way to detect those shifts, reproduce them and decide whether a proposed change actually improves the production distribution.
Trace selection determines what the system learns
Not every production interaction belongs in an evaluation set. Teams need representative successful cases, known failures, rare high-impact tasks and samples from different user groups. Privacy and retention policies should remove or protect personal and confidential content before traces become durable engineering artefacts.
Sampling only low scores can overfit the optimiser to unusual failures, while sampling only frequent tasks can hide safety-critical edges. Dataset versions should record selection rules and time windows. That makes it possible to explain why a change passed last month and failed after traffic or business policy changed.
Agent Optimizer proposes changes, not truth
Microsoft’s Agent Optimizer can use evaluation evidence to suggest improvements to prompts, instructions or skills. The automation reduces the manual cycle of reading logs and guessing which wording might help. It does not eliminate the need to inspect a proposal, because an optimisation can exploit a narrow metric or improve one task by weakening another.
Teams should review diffs just as they review code. Each proposal needs a hypothesis, affected capabilities and a rollback path. High-risk tools deserve stricter approval than a conversational style change. Keeping optimisation runs reproducible also helps engineers understand whether gains came from instructions, model choice, retrieval or a changed tool.
Business criteria must sit outside the editable agent
A particularly strong design point is keeping fixed business criteria outside the prompts and skills being optimised. If an agent can rewrite the evaluator that judges it, the loop can report improvement while moving away from the organisation’s objective. Independent criteria provide a stable contract for accuracy, safety, cost and policy compliance.
The evaluator itself still needs validation. Model-based judges can be inconsistent or biased towards familiar phrasing. Important metrics should combine deterministic checks, expert-labelled examples and calibrated model scoring. Disagreement is useful evidence; it should create a review queue rather than being averaged away.
Regression tests and canaries contain change risk
Microsoft shows the workflow feeding regression tests into GitHub Actions and using canary routing before full rollout. That connects agent iteration to familiar delivery controls. A proposed change must pass the established suite, then prove itself on limited traffic where operators can compare outcomes against the current version.
Canaries need clear stop conditions. Rising tool errors, latency, cost or safety failures should trigger automatic rollback, and deployment operations should be idempotent so recovery does not create duplicate resources or mixed configurations. Version tags must connect each trace to the exact prompt, model, tools and retrieval index that produced it.
Closed-loop does not mean unattended
The approach is best understood as disciplined GenAIOps, not self-improvement without supervision. Humans choose objectives, approve sensitive changes and investigate ambiguous results. Automation handles collection, repeatable scoring and controlled release. That division lets teams iterate faster while retaining accountability for the behaviour exposed to users.
Microsoft Foundry’s pattern is useful beyond one product because it treats agent quality as an operational system. Production evidence creates tests; tests constrain optimisation; deployment controls limit exposure; and new traces start the next cycle. Organisations that build those links can improve agents deliberately instead of accumulating prompt edits whose effects nobody can reliably explain.
Cost and latency belong in the quality contract
Optimisation can improve answer scores while making an agent slower or more expensive. Production criteria should therefore include token use, tool calls, retries and tail latency beside task success. A change that adds several reasoning passes may be justified for a rare high-value workflow but harmful when applied to every customer interaction.
Segmented routing can preserve the trade-off. Straightforward requests can use a fast path, while uncertain or high-risk cases receive deeper reasoning and additional checks. The evaluation loop should test the router as well as the underlying prompts. Misclassification at that boundary can erase the savings or send sensitive work through an underpowered path. Teams should publish a service-level budget for each route and alert when real traffic drifts outside it. They should also retain a small, stable benchmark that prevents a short-term production trend from rewriting the agent around temporary behaviour. Together, these controls keep optimisation tied to customer value rather than one attractive score.