Cohere has published a practical framework for deciding between shared, consumption-based inference and dedicated capacity for its Embed and Rerank models. The central conclusion is that requests per minute are not enough: request shape, traffic pattern, utilisation and latency targets determine the better deployment model.
The analysis is particularly relevant to enterprise search, where the same model can handle large offline indexing batches and tiny interactive queries. Those workloads may show similar request counts while consuming radically different compute and requiring different response times.
Embedding requests come in different shapes
A catalogue-indexing job might send 100 product descriptions of about 200 tokens each in one request. Users are not waiting for an immediate answer, so throughput and price-performance matter most. A search box may send only an eight-token phrase, where latency determines the experience.
Both calls use embedding, but infrastructure tuned for maximum batch throughput may not provide the responsiveness expected by an interactive service. Teams should profile the contents of a typical request rather than classify every call by endpoint name.
Reranking scales with candidate depth
Rerank processes a query against each candidate document. A short query with 50 documents of about 200 tokens can require roughly 11,000 tokens of processing. Increasing the pool from 20 to 100 documents can move the workload from about 4,400 to 22,000 tokens without changing the number of user searches.
Retrieval depth is therefore an infrastructure and product decision. More candidates may improve relevance, but they also increase latency and cost. Evaluation should connect any quality gain with the additional compute required to achieve it.
Steady traffic favours utilisation
Dedicated inference becomes attractive when a service can keep the provisioned capacity busy. A global search product with consistent demand may achieve good utilisation throughout the day. A reporting job with short, concentrated bursts can leave the same hardware idle for long periods.
Consumption pricing fits variable traffic because cost follows use. Dedicated capacity offers predictability and potentially lower unit economics at sustained load. Many systems will need both: provisioned capacity for predictable indexing and shared capacity for fluctuating interactive queries.
Illustrative break-even points vary dramatically
Using representative prices and one dedicated NVIDIA A10 instance, Cohere estimates that short-query embedding may require tens of thousands of requests per minute before dedicated capacity wins. A catalogue batch of roughly 20,000 tokens can cross near 20 requests per minute, while 100,000-token long-document batches can cross around four.
A representative reranking workload with 50 documents crosses near 29 requests per minute. Cohere stresses that these numbers are illustrative, depend on continuous traffic and cannot be compared mechanically across billing models. Different hardware, discounts or working hours shift every threshold.
Maximum throughput is not usable capacity
A benchmark can show how much work an instance completes at saturation, but interactive applications also need a latency service level. Capacity that meets peak throughput only by making users wait is not usable for real-time search.
Production sizing should measure sustainable throughput while holding response time within target. That often means preserving headroom instead of loading hardware to its theoretical maximum. Batch workloads can operate closer to saturation because completion efficiency matters more than per-request delay.
Cost models need real request data
Teams should record tokens per embedding call, number and length of reranking candidates, traffic distribution and latency percentiles. Average request counts can conceal a small number of heavy jobs or brief peaks that dominate infrastructure requirements.
Forecasts should also include growth, redundancy and failure recovery. A dedicated service may need spare instances for resilience, while consumption services may face quotas or variable regional availability during bursts.
Hybrid deployment can match the application
Cohere’s decision sequence is straightforward: understand the request, understand traffic, estimate utilisation and then choose the deployment. It avoids turning a commercial choice into a universal rule.
For enterprise retrieval, the result may be a hybrid architecture. Dedicated capacity can process stable, high-volume indexing while shared inference absorbs unpredictable searches or seasonal reranking. The useful comparison is not the headline unit price, but the cost of meeting the application’s actual quality and latency commitments.
Measure before committing capacity
Organisations considering Cohere Model Vault or shared APIs should replay representative workloads and observe where latency, throughput and cost intersect. The break-even belongs to the workload, not the model name.
That is the broader lesson of Cohere’s analysis. AI infrastructure decisions become more reliable when teams measure what each request carries and when it arrives. Requests per minute remain useful, but only after they are connected to tokens, candidate depth, utilisation and the response time users will accept.
A short measurement period may miss monthly reporting spikes, catalogue rebuilds or seasonal search demand. Capacity planning should cover a representative business cycle and test how both deployment options behave when traffic exceeds the forecast. The cheapest normal-day configuration can become expensive if overflow handling is poorly designed.