Batch speech workloads put economics under pressure

Deepgram has explained how it moved offline batch inference for its Nova-3 speech-to-text model to AWS Trainium2. The company serves more than 200,000 developers and 1,400 organisations and says its systems process quadrillions of audio tokens each year. At that scale, small improvements in accelerator utilisation can materially change the cost of transcribing archives, calls and recorded clinical conversations.

The project targets offline work rather than real-time streaming. Batch transcription can trade interactive latency for higher throughput, packing work efficiently across available hardware. Deepgram reports running eight model replicas on a single trn2.3xlarge instance and achieving between 11 and 70 per cent more audio processed per dollar than its previous options for the tested workload. Those figures should not be generalised to every speech model or traffic pattern.

Static graphs required a deliberate model port

The team used TorchNeuron and torch.compile to turn the encoder and decoder into static graphs for Trainium. Speech models often contain dynamic control flow and variable-length data, so compilation is not simply a matter of changing an instance type. Inputs, batching and execution paths need to be shaped so the accelerator can plan work efficiently without repeatedly falling back to the CPU.

Deepgram explicitly avoided CPU fallbacks, which can hide unsupported operations while adding transfers and unpredictable latency. Engineers compared intermediate layer outputs and word error rate against the established implementation. That two-level validation matters: numerical differences inside a network may be harmless, while a seemingly close tensor can still produce unacceptable transcription errors for names, accents or domain terminology.

Custom kernels close the remaining gaps

Where standard compiled operations were insufficient, the team used the Neuron Kernel Interface Library to write specialised kernels. Custom code can remove a bottleneck and make better use of accelerator memory, but it also becomes part of the model’s maintained software surface. It needs versioning, numerical tests and profiling whenever the model, compiler or instance generation changes.

The sensible optimisation order is to establish a correct compiled baseline, profile representative work and customise only the operations that dominate time or memory. Rewriting many kernels at once can create a fragile fork that is difficult to upgrade. Deepgram’s account is most useful as an engineering pattern: measure, isolate, validate and then tune the small number of operations with a clear economic effect.

A Rust scheduler replaces Python multiprocessing

At maximum throughput, Deepgram found that Python multiprocessing left performance on the table. It built a Rust work-stealing runtime connected through C to the Neuron Runtime, allowing idle workers to take queued work from busy peers. The company reports a further 7.5 per cent throughput increase compared with the multiprocessing implementation under its peak-throughput test.

A custom scheduler adds operational questions beyond model accuracy. Backpressure, cancellation, memory limits and failure isolation need explicit handling. A corrupted audio file or unusually long job should not block an entire device. Operators should record queue time as well as compute time and test recovery when a worker or accelerator process fails midway through a batch.

The result is a workload-specific option, not a universal ranking

Deepgram points to contact-centre analytics and medical transcription as workloads that can benefit from lower-cost offline processing. Both can contain sensitive information, so the surrounding architecture must encrypt source audio, restrict output access and apply retention rules. In healthcare, model evaluation must include specialist vocabulary and the consequences of an incorrect record, not just aggregate word error rate.

The reported gains show Trainium2 can be a credible inference target when a team is willing to compile static graphs, validate every layer and optimise its runtime. Buyers should reproduce the comparison with their own audio lengths, languages, concurrency and quality requirements. Accelerator choice is ultimately an end-to-end question involving model fit, engineering effort, availability and cost. Deepgram’s work demonstrates the payoff that careful co-design can deliver, while also making clear that the result required much more than moving an unchanged container to new hardware.

What teams should test next

Capacity planning should account for compiler time and replica placement as well as steady-state throughput. A configuration that is efficient during a large overnight queue may waste resources during uneven arrivals, while aggressive batching can delay urgent work behind low-priority archives. Separate service classes and queue limits can protect time-sensitive transcription without giving up batch economics. Teams also need a fallback for unsupported operations, regional shortages or a compiler regression. Keeping validated artefacts and a second execution path reduces the risk that infrastructure savings become an availability dependency that operators cannot quickly unwind.

Procurement and engineering teams should record the source date, product version, supported region and evaluation conditions before relying on the result. They should then repeat the same representative tests after a model, compiler, connector or policy changes. Published performance and capability claims are useful starting points, but production confidence comes from evidence collected in the organisation’s own environment. A named owner, measurable acceptance threshold and documented fallback turn a promising technical release into a service that can be governed over time.