Benchmark coverage spans training and agents

NVIDIA has published the first MLPerf results for its Vera Rubin platform, covering model training, post-training and agentic inference. The company says it submitted results across every MLPerf Training 6.1 benchmark and led its Vera Rubin entries, while its first MLPerf Endpoints work showed a more than tenfold generational improvement in throughput per watt for a Kimi K3 agentic workload. The announcement provides an early view of the platform under standardised tests, although some endpoint numbers remain provisional pending MLCommons review.

MLPerf Training measures how quickly a system reaches a defined accuracy target, which is more useful than raw operations per second. The suite spans large language models, image generation, recommendation and reinforcement-learning post-training. NVIDIA reports that Vera Rubin NVL72 delivered up to twice the performance per GPU of an optimised GB300 NVL72 system in selected comparisons. Buyers should inspect the individual submissions, software stack and scale rather than applying the best uplift uniformly to every training job.

The platform also demonstrated scale. On the DeepSeek-V3 benchmark, NVIDIA says Vera Rubin expanded from 256 to 1,024 GPUs with about 97 per cent scaling efficiency, reducing training time from roughly 16 minutes to four. Strong scaling is valuable because adding accelerators only pays when communication and input pipelines keep them productive. Efficiency at one benchmark does not guarantee the same result for a customer model with different sequence lengths, parallelism strategy or checkpoint behaviour.

Scaling efficiency matters at thousands of GPUs

NVIDIA separately ran a GB300 configuration at 10,240 GPUs, while Microsoft Azure matched that scale with GB200 systems. Large submissions show that networking and software can coordinate thousands of devices, but they represent infrastructure far beyond most organisations. The relevant question for a buyer is how performance changes at the cluster size it can actually obtain and keep utilised. Small-scale efficiency, queue availability and the ability to share capacity may matter more than a record at the largest configuration.

MLPerf Endpoints is designed around complete agentic work rather than isolated token generation. Agents reason, call tools and verify results, so a useful benchmark must consider whether they finish tasks correctly while serving concurrent users. It also needs to balance time to first token, ongoing interactivity and total throughput. This is a welcome shift from quoting peak tokens per second without acknowledging that a slow or inaccurate agent may deliver little business value.

Endpoint tests introduce work-per-watt metrics

NVIDIA’s provisional Kimi K3 result claims more than ten times the throughput per kilowatt of GB300 at a shared response-speed threshold. The Vera Rubin run used 32 accelerators and much higher concurrency than the 48-accelerator GB300 comparison, with operating power also higher in absolute terms. Throughput-per-watt is the relevant ratio, but different concurrency and latency characteristics require careful reading. The result is subject to peer review and should not be treated as final until MLCommons verifies it.

Power efficiency is becoming a binding infrastructure constraint. Data centres may have accelerator supply but insufficient electrical or cooling capacity to deploy it. Measuring completed agent work per watt can connect hardware design to the number of users or tasks a facility can support. Operators should still include CPUs, networking, storage and cooling in facility-level energy analysis. Accelerator operating power alone does not equal the total energy consumed by a production service.

Provisional results need careful interpretation

Benchmark transparency makes these claims more useful than an internal chart, because submission records expose systems and rules for comparison. Customers should reproduce a representative workload where possible and test software maturity, failure recovery and sustained operation. Early hardware can post strong benchmark numbers while production tooling, availability and support continue to mature. Continuous software optimisation may improve performance, as NVIDIA expects, but it can also change the comparison over time.

Vera Rubin’s MLPerf debut suggests meaningful gains in training density and agentic throughput, particularly where power is scarce. It does not make procurement a one-number decision. Australian operators should compare validated task completion, latency, energy, regional capacity and total system cost against existing fleets. The most valuable contribution of the endpoint benchmark may be its framing: infrastructure should be judged by correct work delivered to users, not by theoretical arithmetic or tokens generated without regard to the outcome.

For service designers, the endpoint submission also reinforces that concurrency is a product choice. Driving more simultaneous work can improve aggregate throughput while degrading the latency experienced by an individual user. Capacity models should therefore apply the organisation's own response-time and accuracy thresholds, then report completed tasks per dollar and per kilowatt at those limits. That translation is necessary before a benchmark result becomes a defensible purchasing forecast. Teams should repeat the exercise after model, quantisation or serving-software changes, because the most efficient production configuration can shift without any hardware change.