The incumbent model had written its own answer key
Property Finder discovered that 96.8 per cent of labels in a vision-model evaluation set had been copied from the predictions of its incumbent system rather than independently verified by people. Any alternative was therefore judged by how closely it reproduced the existing model, not by whether it correctly understood a property image. AWS calls this circular evaluation a structural source of bias.
The dataset also contained concrete errors, including hallucinated watermark text, missed characters and single room labels for images that clearly showed several spaces. Property Finder processes tens of thousands of images across Middle Eastern and North African listings, assessing eleven attributes such as room type, furnishing, construction state, watermarks and image quality.
Independent annotation changed the model ranking
A domain expert relabelled 356 images from the United Arab Emirates without seeing earlier predictions, while the AWS Generative AI Innovation Center reviewed ambiguous cases. The first 256 images became a development set and a later 100 were added for final comparison. The team also moved appropriate tasks from single-label to multi-label scoring and reported Micro F1, Macro F1 and per-attribute measures.
The group then evolved prompts through error analysis and automated optimisation. Llama 4 Maverick 17B Instruct on Amazon Bedrock rose from a Micro F1 of 0.75 with the shared prompt to 0.81 on the development set. AWS reports 0.80 on the final 356-image set, compared with 0.56 for the proprietary baseline, although the development images were included in that final set.
Prompt evolution improved accuracy and economics
Property Finder selected the Llama model after comparing accuracy and cost. AWS estimates US$2.35 per 1,000 images, 86 per cent below the previous setup, and reports a 40–60 per cent latency reduction in production. These figures depend on token assumptions, regional pricing and the workload’s prompt design, so other organisations should reproduce the calculation with their own traffic.
The deeper lesson is that an evaluation can look rigorous while rewarding historical mistakes. Teams should trace every label to its origin, keep an untouched test set and use domain experts for consequential ambiguity. Model disagreement can help prioritise samples for review, but no candidate model should be allowed to define the ground truth against which its competitors are judged.
What teams should validate before production
For implementation, teams should record the exact model, software, dataset, connector and policy versions behind every result. Representative evaluation must include normal work, boundary cases, deliberate misuse and component failure, with outcomes separated by user role, data sensitivity and consequence. A strong average should never hide a serious failure in a critical path. Changes should pass the same regression set before promotion, and operators need a documented fallback that preserves evidence rather than silently broadening access or lowering quality.
Procurement and planning should account for the human work required to review exceptions, maintain controls and investigate incidents. Published performance or efficiency claims are useful starting points, but production confidence comes from evidence collected in the organisation’s own environment. A named owner, measurable acceptance threshold, cost model and tested shutdown path turn a promising release into a service that can be governed over time. Where decisions affect safety, rights, money or regulated records, accountable professional review remains essential.
Security and operational evidence
Security review should map every identity, credential, network destination, stored artefact and administrative access path involved in the workflow. Logs need enough context to reconstruct who initiated an action, which evidence was consulted, what the system proposed and what ultimately executed. Sensitive prompts and traces should not be retained merely because they are useful for debugging. Retention, redaction and access must follow the underlying data classification, with a practical process for correcting or removing material when policy or law requires it.
Operational monitoring should measure quality and service behaviour together. Latency, throughput and availability matter, but so do unsupported claims, missed constraints, unsafe tool calls and the rate at which people overturn recommendations. Teams should rehearse dependency outages and partial failures before unattended operation. If a model, connector or external service becomes unavailable, the workflow should fail predictably, preserve its state and tell the user what remains unfinished. Clear degradation is safer than an agent improvising around a missing control.
Transparency should scale with adoption
Organisations should explain the scope and limitations of the deployed capability to the people affected by it. Users need to know when automation is involved, how to challenge an outcome and where a human can intervene. That transparency improves feedback and prevents a pilot result from being interpreted as blanket approval for broader autonomy. Expansion should follow demonstrated evidence, not novelty or pressure to match a competitor announcement. The durable advantage comes from dependable integration, careful measurement and learning that survives the next model or product update.
Finally, teams should schedule a review after real usage rather than treating launch as the end of governance. The review should compare expected benefits with observed quality, cost, incidents and user behaviour, then retire workflows that do not justify their complexity. A service that works well can expand through the same evidence-based process. A service that fails should be narrowed or stopped without losing the records needed to understand why. This feedback loop is what turns experimentation into accountable operational capability.