AWS has outlined a new evaluation pattern for production AI agents: use small, specialised decision models for narrow checks, then escalate uncertain or complicated cases to broader evaluators. The implementation connects decision-model backends to Amazon Bedrock AgentCore Evaluations through code-based evaluators, giving teams calibrated probabilities rather than relying on free-form model judgements for every test.
The 4 November technical guide uses two examples. Jev is a hosted decision model from TypeSafe, while Strands Decider 2B is an open-source two-billion-parameter model that teams can run themselves. Both can answer tightly framed questions about an agent turn or session and return a label with a confidence distribution.
A narrower tool for repeatable questions
Many agent-evaluation questions are constrained. A team may want to know whether a response stayed on topic, whether a tool call is safe to approve, or whether a claim is grounded in retrieved material. A general-purpose LLM can assess those questions and explain its reasoning, but that flexibility is not always necessary.
A decision model instead chooses from a defined answer set. AWS describes yes-or-no questions, choices among several categories and ordered scoring rubrics. Because the output includes a probability distribution, developers can establish thresholds, compare calibration against human labels and route low-confidence cases for deeper review.
This does not replace the built-in evaluators in AgentCore. AWS positions decision models as an additional layer. AgentCore Evaluations can already score agents during development, in batch jobs using stored sessions and online against production traffic. It includes built-in evaluators and supports custom LLM-as-a-judge or code-based evaluators. The decision model is called from the code-based path.
How the reference workflow is assembled
The example centres on an HR assistant deployed to AgentCore runtime. It answers questions using fixed mock tools for leave balances, requests, policies, benefits and pay information. The controlled data makes it easier to compare evaluator results with known facts and labelled scenarios.
The base deployment packages deterministic checks in AWS Lambda, registers them as code-based evaluators, and combines them with built-in correctness, helpfulness and relevance metrics. Optional flags add Jev or Strands Decider evaluators for groundedness, helpfulness and goal completion. The adapter receives OpenTelemetry spans, formats the relevant conversation state for the model and translates its probability output into the result shape AgentCore expects.
The three question types serve different purposes. A binary question can report the probability that an answer is grounded. An ordered score can estimate helpfulness across a four-level rubric. A multi-way choice can classify whether a goal was completed, partly completed or not completed. These outputs can be aggregated programmatically without parsing an explanation.
Calibration has to come before automation
A key strength of the AWS guidance is that it does not treat a confident number as proof. Rewriting a rubric as a decision question creates a new measurement, and that measurement needs validation on the organisation’s own traffic.
AWS recommends building a labelled sample that includes failures and edge cases, collecting independent human labels, then running the decision model and the corresponding built-in evaluator on identical inputs. Teams should compare agreement with the human majority, examine whether higher confidence actually correlates with higher accuracy, and read the cases where evaluators disagree.
Split human votes are particularly useful because they reveal ambiguity in the rubric itself. If model errors cluster around those cases, the problem may be the question rather than the evaluator. AWS also advises rerunning the exercise whenever a question or model version changes.
A cascade can lower evaluation cost
The most practical deployment pattern is a cascade. A fast decision model screens every turn and emits a calibrated pass-or-review signal. Sessions below the chosen threshold then go to a more capable LLM evaluator, which can connect evidence and explain what went wrong. Human review can sit above both for consequential decisions.
This arrangement gives teams a way to expand coverage without paying the latency and inference cost of a frontier evaluator on every interaction. It can also make monitoring more operationally useful because the first stage returns stable labels and scores suitable for dashboards, alerts and release gates.
The trade-off is scope. Decision models work best when the question is precise and the answer space is fixed. They are poorly suited to assessments that require subtle interpretation, broad context or a detailed explanation. Production teams will need to decide which checks are genuinely narrow and retain richer evaluation for the rest.
For Bedrock users, the guide turns that design into a concrete integration path. The combination of OpenTelemetry traces, Lambda-based adapters and AgentCore’s evaluator framework lets organisations introduce specialised models without rebuilding their entire evaluation pipeline.
Operationally, teams should record the evaluator version, threshold and question text beside every result. That makes later comparisons meaningful and avoids treating scores from changed rubrics as one continuous metric. With those controls, the approach can support release gates, continuous production monitoring and targeted sampling for human quality review.