News
Updates, product notes and practical coverage for AI platforms and operators.
Google Cloud has published a five-rule framework for evaluating agent skills, arguing that one successful prompt is not adequate evidence. The guidance combines linting, vocabulary-overlap checks, repeated with-and-without-skill trials, security analysis and separate measures for correctness, efficiency and safety. It also describes optimisation experiments where candidate skills attempted to game their graders.
AWS has published a nine-stage evaluation framework for production agents built with Amazon Bedrock AgentCore. Its internal sales system exposed fluent but serious data errors that ordinary availability monitoring missed. The framework combines pre-release tests, deployment gates, canary scoring, production sampling, drift detection and feedback so quality is measured continuously rather than at launch alone.
Google Cloud has distilled four evaluation-engineering lessons from building its Developer plugin. The guidance argues that agent tests should inspect tool calls and execution traces, isolate infrastructure failures from model failures, use deterministic checks where possible and treat evaluation suites as maintained software rather than one-off launch gates.
Anthropic has given six METR evaluators employee-like access to internal risk systems, company hardware and office space. METR can publish assessments that Anthropic cannot veto, while narrow predefined exclusions protect customer data, privileged material and model architecture. SecureBio and Accenture are also evaluating Anthropic, with Hugging Face discussions underway.
Cohere has introduced RCP-nDCG@10, an AI-assisted method for evaluating enterprise retrieval when traditional relevance labels are incomplete. A 46-person blind study found reviewers preferred its conflicting system choice 70 per cent of the time. Cohere has released supporting code and data and used the method while developing Embed 5.
TypeSafe AI says it will avoid standard public benchmark tables in model releases, publish dated one-off evaluations and disclose caveats, cherry-picking and unfavourable evidence. The policy, outlined before its Jev launch, responds to benchmark optimisation and ranking feedback loops while placing more responsibility on customers to run private, workload-specific tests.
Braintrust now supports native tracing for agents running on Cloudflare, using OpenTelemetry data from the Agents SDK and related packages. Developers can inspect model calls, tools, sub-agents, tokens and Workers infrastructure in one trace, then turn production activity into evaluation datasets. Version and runtime prerequisites apply.