The expensive bottleneck may sit outside the GPU
Google Cloud is making the case that AI infrastructure economics depend as much on storage as accelerators. GPUs and TPUs can represent most of a project’s capital or committed cloud spend, yet they deliver no useful work while waiting for training data, checkpoints or model weights. Google’s latest infrastructure guidance connects Cloud Storage caches, Parallelstore and Managed Lustre into workload-specific paths intended to keep accelerators occupied. The central lesson is sensible: buying faster compute without measuring the data path can increase cost without improving time to result.
Training and agent workloads create several different I/O patterns. Large initial dataset reads benefit from locality and aggregate throughput. Synchronous checkpoints generate intense bursts from many workers at once. Inference fleets may repeatedly load the same weights, while agent systems can generate large volumes of small state and retrieval operations. No single storage tier is ideal for all of them. Architecture should begin with traces of file size, read-write ratio, concurrency, reuse and recovery requirements rather than a generic recommendation to put everything near the accelerator.
Google positions Cloud Storage Anywhere Cache for zonal access to frequently reused object data and Rapid Cache for managed high-throughput caching across broader footprints. A cache can stabilise read peaks and reduce repeated origin traffic, but it introduces consistency and eviction behaviour that applications must understand. Teams should test cold-start performance, failure of the cache layer and the effect of a working set larger than capacity. A high hit rate in a steady benchmark does not guarantee that a rapidly changing training corpus will behave similarly.
Different access patterns need different storage
For file-oriented workloads, Parallelstore provides a managed parallel file system based on DAOS, while Google Cloud Managed Lustre targets demanding AI and HPC jobs. Google states that Managed Lustre can deliver up to 10 TB/s per instance and absorb the bursts created by synchronous checkpointing and reinforcement learning. Maximum platform figures are not application guarantees. Directory layout, client count, network placement and checkpoint format can all prevent a workload from reaching headline throughput. Benchmarking must use the real model and data pipeline.
A new Dynamic Tier for Managed Lustre is intended to place hot and cold data within one namespace, reducing the performance cliff of a manually managed cache. Cloud Storage Mirror can synchronise datasets in the background so a cluster does not wait for scripts to copy files immediately before training. Automation removes repetitive work, but operators still need visibility into replication lag and failure. A job should know whether the required version is local and complete rather than assuming that a mounted path means the data is ready.
Published customer results need local validation
Google cites production results from several customers. Thinking Machines Lab reportedly sustained read peaks above 1.8 TB/s with an 89 per cent cache hit ratio and eliminated rate-limiting surges. Shopify deployed more than 50 Rapid Caches, reporting peaks above 2 TB/s and latency reductions of up to 90 per cent from its baseline. Salesforce AI Research uses Managed Lustre as an external cache to keep GPUs busy. These examples demonstrate possibility, not a forecast for every architecture.
The right business metric is not storage cost in isolation. Teams should track accelerator utilisation, training completion time, failed-job recovery, checkpoint overhead and cost per successful experiment. A more expensive storage layer may reduce the total bill if it shortens reserved GPU time. Conversely, provisioning maximum throughput for a pipeline that spends most of its time in preprocessing can waste money. FinOps dashboards should connect storage and accelerator charges to the same job identifier so trade-offs become visible.
Utilisation is an end-to-end system metric
Resilience matters alongside speed. Checkpoints exist so a long run can recover, which means they need durable placement and tested restore procedures. A cache or parallel file system should not become the only copy of a valuable model state. Teams must define when data moves to durable object storage, how integrity is verified and which recovery-point objective the pipeline actually meets. They should rehearse the loss of a node, zone or cache during a representative job rather than relying only on service availability claims.
Australian AI teams should also verify regional availability, quota and data-residency implications for each storage product. Moving a dataset to a distant region to reach a preferred accelerator can create governance and transfer-cost problems that outweigh a performance gain. Google’s portfolio gives architects useful building blocks, but accelerator return on investment remains an end-to-end property. The winning design will keep compute supplied, preserve reproducibility and recovery, and scale down when unused without turning storage optimisation into another opaque layer of infrastructure.
Capacity planning should also include the cost of unpredictability. A pipeline that performs well only after its cache is warm can miss deadlines when a new dataset arrives or a cluster moves zones. Teams should publish cold, warm and recovery benchmarks together, then set service objectives for each phase. That exposes whether storage is genuinely supporting repeatable AI delivery or merely producing an impressive best-case graph.