AWS adds checks for whether AI agents choose and follow the right skills

A plausible answer can hide the wrong path

AWS has described new evaluators for agents that use reusable skills. Its AWS announcement, published on 22 September under Announcements, focuses on two failures that a conventional answer-quality score may miss: choosing an inappropriate skill and choosing the right skill but ignoring part of its procedure. The distinction matters when a result looks fluent while the agent has skipped a required check.

A skill can package instructions, tool connections, knowledge and workflow rules. In an HR example, an assistant might select a benefits playbook when a leave-planning skill was needed. Alternatively, it could select the leave skill but submit a request without checking rollover policy. The final sentence to the employee might look reasonable in either case, yet the underlying process would be different.

AWS positions Strands Evals and Amazon Bedrock AgentCore Evaluations as ways to inspect that process. This is not a claim that an evaluator can certify every agent as safe. It is a more specific measurement of how an agent used a skill, which teams can combine with their own task outcomes and human review.

What the three measurements ask

Skill Selection Accuracy scores whether an invoked skill fits the task. Skill Instruction Following scores how completely the agent carried out the skill’s prescribed steps. Both are available in Strands Evals and AgentCore Evaluations, according to AWS. The first is a binary judgement for each invoked skill; the second uses five levels from fully followed to not followed.

Strands Evals also provides Skill Invoked, a deterministic check that a named skill was loaded. It does not call a judge model. That makes it useful for a regression case where a particular route must be taken. It should be paired with selection and instruction-following checks, because loading a skill alone does not show whether the skill was appropriate or used correctly.

Separating the scores gives a developer a clearer repair path. Poor selection may indicate overlapping skill descriptions or ambiguous routing. Poor instruction following may point to unclear steps, missing tool access or a model struggling with the workflow. A single overall pass rate could conceal these causes and lead a team to tune the wrong component.

From test cases to production traces

For development, Strands Evals can work with a recorded agent trajectory, including messages and tool actions. A team can define representative tasks, rerun them and make a critical skill check part of a release gate. AWS says its extraction recognises several agent environments, but teams should confirm that the particular trace format in their system captures skill use accurately.

AgentCore Evaluations can use OpenTelemetry traces from agents running on AgentCore or elsewhere. AWS describes on-demand, batch and continuous-sampling modes. Each skill-level result can be associated with the relevant session, trace and span. That link is important: a score without the underlying run is difficult to investigate when a production agent makes a surprising choice.

Judge-based scores still require calibration. A broad skill with six unrelated responsibilities is difficult to evaluate fairly, and a rubric must distinguish a required step from a branch the agent did not need to take. Teams should compare sample scores with expert review before treating a threshold as a dependable deployment gate.

Using the results responsibly

A sensible evaluation set includes straightforward routing, overlapping skills, missing data and cases where the safest answer is to stop or escalate. For each case, record the expected skill, the required steps and the actual outcome. A binary routing pass cannot replace evidence that the underlying business action was correct. It should sit beside factuality, security and user-experience checks.

Production monitoring also needs restraint. Detailed traces can contain sensitive user material or tool results. Organisations should set retention, access and redaction controls before collecting them at scale. The benefit of a fine-grained evaluator is strongest when a team can inspect a failure without creating an unnecessary archive of private conversations.

AWS’s addition makes agent evaluation less dependent on how polished the last answer appears. It gives builders a way to ask whether the agent chose the right procedure and followed it. If teams use those measurements to improve skill boundaries and verify real task outcomes, the result could be more reliable agents; the new metrics by themselves are evidence, not a guarantee.

The operational response to a failure should also be specific. If selection is wrong, teams can revise skill descriptions and test confusing pairs. If steps are skipped, they can simplify the procedure, add checks or reduce the agent’s authority until it performs consistently. A generic instruction to be more careful provides little basis for measuring whether the problem has been fixed.

Versioning matters because skill content changes. A passing result for one edition of a playbook should not automatically approve a rewritten edition with new tools or permissions. Keep the evaluator case, rubric, skill version and representative trace together. This turns a one-off score into evidence that can support a later review of how the agent behaved.