Baseten has made NVIDIA’s Nemotron 3.5 ASR Streaming models available through its model library, adding a deployable option for low-latency speech recognition. The release includes an English model and a multilingual model covering 40 language locales, both delivered through NVIDIA NIM.
The models are aimed at workloads where transcription needs to arrive while someone is still speaking: voice agents, live captions, contact-centre assistance and meeting products. For these applications, accuracy is only one part of the experience. The system also has to support many persistent connections and finalise each segment quickly enough for the next component to respond.
A compact streaming architecture
Each model has 600 million parameters and uses a FastConformer-RNNT design with a 24-layer encoder and an RNNT decoder. The multilingual version accepts a language identifier prompt, giving an application a way to select the expected locale rather than relying entirely on automatic detection.
The output includes punctuation and capitalisation, which reduces the post-processing needed before a transcript is shown to a user or passed to a language model. The models are released under NVIDIA’s Open Model Development and Use Licence Agreement, version 1.1; teams considering redistribution or a specialised deployment should review those terms for their use case.
Baseten tests concurrency on one H100
In Baseten’s testing, a single NVIDIA H100 GPU supported as many as 100 concurrent real-time WebSocket streams. Reported finalisation latency ranged from 98 milliseconds with eight streams to 138 milliseconds with 100 streams. The company also says gRPC can provide higher throughput and that Nemotron delivered up to six times the throughput of Parakeet RNNT in its comparison.
These are vendor-run measurements, not a guarantee for every deployment. Audio duration, chunk size, language mix, networking, batching and the surrounding application can all affect capacity and latency. Teams should reproduce the test with representative calls and include end-to-end response time, not only inference time.
Baseten reports an average word error rate of 8.84 per cent across 19 supported languages in the FLEURS benchmark. That aggregate is useful for orientation, but it can hide substantial variation by language, accent and acoustic setting. Domain vocabulary and noisy telephone audio may produce different results from a benchmark dataset.
Deployment is the differentiator
Nemotron 3.5 ASR Streaming is not a new model family in isolation; the practical announcement is its availability as a deployable model on Baseten. NVIDIA NIM packages the inference service, while Baseten supplies the model-library workflow and production infrastructure around it.
That can shorten the path from a model evaluation to an endpoint, especially for teams that do not want to assemble the serving stack themselves. Before adopting it, buyers should confirm regional GPU availability, autoscaling behaviour, idle capacity, observability and how audio data is retained or logged.
Streaming systems also need graceful handling for connection loss, partial transcripts and speaker turns. A strong technical trial should measure time to first text, time to final text, correction frequency and cost per audio hour under realistic concurrency. The release gives developers another serious speech-recognition option; production fit will be decided by those workload-specific results.