Overview

Nemotron-3-Diarization is an open-weight speaker diarization model from NVIDIA that identifies who spoke when in real-world audio. Its most important distinction is that one checkpoint supports offline processing and streaming operation with up to eight speakers, with configurable output resolution in multiples of 10 ms and chunked inference that removes a maximum-duration limit.

The model follows Sortformer and orders output channels by each speaker’s first arrival, which resolves the speaker-permutation problem without requiring fixed speaker identities in advance. Streaming uses an Arrival-Order Speaker Cache and FIFO queue to carry speaker information and recent frame context between chunks.

The provided information does not state the parameter count, training dataset, training steps, GPU VRAM requirement, or inference throughput, so those values should not be assumed. NVIDIA NeMo Speech provides the primary Python framework, while NeMo-Speech.cpp provides a lightweight native C++ runtime. The model uses the openmdw-1.1 license and is described as ready for commercial and non-commercial use.

Best Use Cases

Real-time meeting diarization. Use the model to label speakers during meetings, interviews, calls, and live events. Its streaming design preserves speaker identities across chunks through the AOSC and FIFO mechanisms, while the recommended low-latency configurations provide 1.04 s, 0.64 s, or 0.32 s of input-buffer latency before computation time.

Long-form offline transcription with speaker labels. Use it to segment recordings and assign speaker indices before combining the results with an ASR transcript. Chunked inference supports audio durations without a stated maximum, and the offline-style configuration uses a 30.4 s input buffer for processing that favors context over responsiveness.

Word-level speaker-attributed transcripts. The NeMo-Speech.cpp runtime can combine transcription and diarization with nemo-speech transcribe meeting.wav --diarize --json. This fits meeting archives, interview analysis, call-center review, and searchable media workflows where each word needs a speaker tag rather than only time-stamped speaker segments.

Diarization for recordings with as many as eight speakers. The model is designed for conversations that exceed the four-speaker limit of the related Streaming Sortformer 4spk model. Its eight output channels support larger meetings and panel discussions, though the supplied information does not provide accuracy results for each speaker count.

Custom latency and evaluation studies. Researchers can vary chunk_len, fifo_len, chunk_right_context, and spkcache_update_period, then measure diarization error rate and speed with NeMo’s e2e_diarize_speech.py example. The model returns speaker-activity probability tensors, which support downstream thresholding and analysis.

Limitations

The model supports at most eight speakers. It should not be selected for sessions that require more than eight simultaneous or represented speaker channels without a separate strategy for splitting, clustering, or post-processing the audio.

The latency figures describe input-buffer latency only. They equal (CHUNK_LEN + RIGHT_CONTEXT) × 80 ms and exclude model computation time, device transfer, audio decoding, and application overhead. The 80 ms minimum mentioned for a single checkpoint is not presented as the lowest recommended operating point; the README identifies 0.32 s as the lowest recommended configuration. Ultra-low latency can reduce available context and may affect diarization quality, but the supplied material does not quantify that quality tradeoff.

The documentation does not provide DER scores, benchmark datasets, real-time factors, throughput, memory use, GPU requirements, or parameter count. You must measure these values on your target hardware and audio domain. The recommended NeMo setup requires Python 3.12 or later, Cython, a recent PyTorch version, libsndfile, ffmpeg, and nemo-toolkit[asr]. The native C++ route may reduce runtime integration overhead, but the provided information does not state its hardware requirements or speed.

Speaker indices represent arrival order, not persistent identities such as “Alice” or “Bob.” A speaker who appears first receives the first output-channel position. Applications that need named identities must add enrollment, recognition, or another identity-mapping layer.

The model card does not list known demographic biases, noise-specific failure rates, overlap-speech metrics, microphone constraints, or domain-specific accuracy limits. Do not treat the absence of reported issues as evidence that the model performs equally across accents, reverberation conditions, crosstalk, far-field microphones, or noisy recordings.

The openmdw-1.1 license is the stated license. The card also says commercial and non-commercial use are allowed. Review the license text and your compliance requirements before deployment; the supplied material does not enumerate additional obligations.

How it Compares

Nemotron-3-Diarization-preview

Choose Nemotron-3-Diarization when you need the documented production-oriented checkpoint with streaming and offline operation, up to eight speakers, configurable latency, and the AOSC/FIFO streaming design. Choose the preview model when you need to evaluate the earlier or preview release described in its separate overview, but the supplied information does not provide enough benchmark data to claim that either model is faster, cheaper, or more accurate. The practical tradeoff is documented release maturity and feature clarity versus the uncertainty of a preview artifact.

diar_streaming_sortformer_4spk-v2.1

Choose Nemotron-3-Diarization for conversations requiring up to eight speakers and for a single checkpoint that covers offline-style processing plus several streaming latency configurations. Choose Streaming Sortformer Diarizer 4spk v2.1 when four speakers are sufficient, and its documented 117M-parameter FastConformer–Transformer architecture fits your deployment or benchmark baseline. The supplied comparison data does not include speed, cost, or DER, so the concrete known tradeoff is speaker capacity: eight speakers here versus four in the alternative.

diar_sortformer_4spk-v1

Choose Nemotron-3-Diarization when you need streaming support, AOSC/FIFO state management, and up to eight speakers. Choose Sortformer Diarizer 4spk v1 for a four-speaker offline Sortformer baseline with a documented 123M-parameter FastConformer–Transformer architecture. The alternative is older and limited to four speakers in the supplied description; no reliable speed, cost, or quality ranking is provided.

nemotron-3.5-asr-streaming-0.6b

Choose Nemotron-3-Diarization for speaker segmentation and speaker-attributed audio; it is a diarization model rather than an ASR model. Choose nemotron-3.5-asr-streaming-0.6b for streaming speech recognition if you have access, but the supplied notice says it is under internal evaluation and available only to NVIDIA employees. These models solve different tasks, so neither is a direct quality or speed substitute for the other.

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

Choose Nemotron-3-Diarization for dedicated, low-latency speaker activity and speaker-index output. Choose Nemotron 3 Nano Omni for broad multimodal workflows involving video, audio, images, text, GUI understanding, OCR, transcription, and enterprise Q&A. The alternative is a general multimodal model rather than a specialized diarizer, and the supplied information gives no basis for comparing diarization accuracy, latency, or operating cost.

Technical Specifications

• Task: Speaker diarization and voice-activity-style frame classification.
• Architecture family: Sortformer; the streaming design uses Arrival-Order Speaker Cache and FIFO context.
• Speaker capacity: Up to 8 speakers.
• Speaker ordering: Output channels follow each speaker’s first arrival in the input audio.
• Inference modes: Offline and streaming.
• Streaming state: AOSC speaker cache plus FIFO queue.
• Input-buffer latency: A single checkpoint supports latency as low as 80 ms; the lowest recommended configuration is 0.32 s.
• Recommended configurations:
• Very high latency/offline: 30.4 s; SPKCACHE_LEN=264, FIFO_LEN=40, CHUNK_LEN=340, RIGHT_CONTEXT=40, UPDATE_PERIOD=300.
• Low latency: 1.04 s; 264, 264, 9, 4, 222.
• Very low latency: 0.64 s; 264, 264, 6, 2, 222.
• Ultra-low latency: 0.32 s; 264, 264, 3, 1, 222.
• Streaming unit: Parameters are measured in 80 ms frames.
• Output resolution: Configurable in multiples of 10 ms.
• Maximum duration: No stated maximum with chunked inference.
• Frameworks: NVIDIA NeMo Speech, Transformers, and NeMo-Speech.cpp.
• NeMo Python class: SortformerEncLabelModel.
• Transformers class: AutoModelForAudioFrameClassification.
• Model processor: AutoProcessor.
• Model identifier: nvidia/Nemotron-3-Diarization.
• Model file loading: Hugging Face from_pretrained; local .nemo files through restore_from.
• Evaluation: NeMo e2e_diarize_speech.py; supports DER and speed evaluation across streaming settings.
• License: openmdw-1.1.
• Commercial use: The model card states commercial and non-commercial use are permitted.
• Release date: September 23, 2026.
• Downloads listed: 4,282.
• Parameter count, training data, training compute, VRAM, quantization, and throughput: Not provided.

Model Inputs and Outputs

Inputs

• Audio file path: A single path such as /path/to/multispeaker_audio1.wav.
• List of audio paths: Multiple files can be processed in a batch.
• NumPy array: A single float32 waveform or a list of arrays.
• NumPy sample rate: The default is 16,000 Hz; when using arrays, pass the correct integer sample_rate.
• Manifest path: A line-delimited JSON file with audio_filepath, offset, and duration.
• Streaming audio: Audio arrives chunk by chunk through the Transformers streaming processor.
• Streaming modes: "low_latency", "very_low_latency", and "ultra_low_latency".
• Final streaming chunk: Set is_last_audio_chunk=True so the processor scores remaining frames without look-ahead.
• Batch size: The examples use batch_size=1 for one file and batch_size=2 for two NumPy arrays; the evaluation example uses batch_size=32.

Outputs

• Speaker segments: A list containing begin_seconds, end_seconds, and speaker_index.
• Speaker probabilities: Optional tensors from include_tensor_outputs=True.
• Transformers logits: Shape (1, num_frames, 8), with one frame every 10 ms in the offline example.
• Speaker dictionaries: processor.extract_speaker_dict(logits, inputs.attention_mask) returns speaker, start, and end fields.
• Streaming state: The Transformers forward pass returns speaker_cache, which must be passed to the next call.
• Speaker identity: Indices reflect arrival order, not names.
• Transcription integration: NeMo-Speech.cpp can emit word-level speaker tags in JSON.

Getting Started

Install the NeMo dependencies, then run:

apt-get update && apt-get install -y libsndfile1 ffmpeg
uv pip install Cython packaging
uv pip install 'nemo-toolkit[asr]'
from nemo.collections.asr.models import SortformerEncLabelModel

model = SortformerEncLabelModel.from_pretrained(
    "nvidia/Nemotron-3-Diarization"
)
model.eval()

model.sortformer_modules.chunk_len = 340
model.sortformer_modules.chunk_right_context = 40
model.sortformer_modules.fifo_len = 40
model.sortformer_modules.spkcache_update_period = 300
model._check_streaming_parameters()

segments = model.diarize(
    audio=["/path/to/meeting.wav"],
    batch_size=1,
)

for segment in segments[0]:
    print(segment)

For NumPy input:

import numpy as np

audio = np.random.randn(16000 * 10).astype(np.float32)

segments, probabilities = model.diarize(
    audio=[audio],
    batch_size=1,
    sample_rate=16000,
    include_tensor_outputs=True,
)

For a local .nemo checkpoint:

model = SortformerEncLabelModel.restore_from(
    restore_path="/path/to/Nemotron-3-Diarization.nemo",
    map_location="cuda",
    strict=False,
)
model.eval()

Frequently Asked Questions

Q: Can I use Nemotron-3-Diarization commercially?

A: Yes. The model card states that it is ready for commercial and non-commercial use. The listed license is openmdw-1.1.

Q: How many speakers can Nemotron-3-Diarization handle?

A: It supports up to eight speakers. Its Transformers offline output has eight speaker channels.

Q: What latency should I expect?

A: The recommended input-buffer settings are 1.04 s, 0.64 s, and 0.32 s for low, very low, and ultra-low latency. These values exclude computation time; end-to-end latency depends on hardware and runtime.

Q: Does the model require a GPU or a specific amount of VRAM?

A: The supplied documentation does not state a GPU requirement, VRAM requirement, or inference speed. The NeMo local-checkpoint example uses map_location="cuda", but that example does not establish a minimum GPU specification.

Q: What audio formats can I provide?

A: NeMo accepts file paths, lists of file paths, NumPy arrays, lists of arrays, and line-delimited JSON manifests. For NumPy input, pass the correct integer sample rate; the documented default is 16,000 Hz.

Q: Can I process an unlimited-length recording?

A: Chunked inference has no stated maximum audio duration. The model processes the recording in chunks rather than requiring the entire recording in one input.

Q: Can I fine-tune this model?

A: The documentation states that NVIDIA NeMo Speech supports training, fine-tuning, and inference. It does not provide a fine-tuning recipe, dataset requirements, or compute requirements.

Q: Does it produce speaker names?

A: No. It produces speaker indices ordered by each speaker’s first arrival. Mapping those indices to names requires a separate speaker-identification or enrollment system.

This is a simplified guide to an AI model called Nemotron-3-Diarization maintained by NVIDIA. If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter.