For robots such as voice assistants, embodied agents, and AI tutors, the ability to respond in natural language unlocks a fundamentally different experience for users: one that's intuitive, accessible, and doesn't require users to learn new interfaces and notations. Achieving truly natural language in this context is difficult. Users notice robotic prosody, unnatural pauses, and voices that don’t match the context. Getting speech synthesis right is the difference between a tool that people tolerate—or even work around—and one they may actually want to use. For the U.S. Department of War, speech synthesis can be a powerful force multiplier, supporting the use of tools that enhance situational awareness, help manage cognitive loads, and facilitate multinational collaboration. AI text-to-speech systems can now deliver near-human prosody, real-time performance, and context-aware emotional nuance—making synthetic voices that feel truly alive, responsive, and trustworthy.

This post dissects four open-source text-to-speech (TTS) systems that represent distinct points in the design space: NeuTTS Air (LLM + neural codec, excellent zero-shot cloning), Piper (VITS-based, blazing fast, runs anywhere), VibeVoice (σ-VAE + diffusion, built for long-form multi-speaker content), and Chatterbox (Llama backbone + HiFi-GAN, with paralinguistic control). Rather than declaring a winner, we map out where each architecture shines. Along the way, we build intuition for shared building blocks (phonemizers, mel spectrograms, vocoders, tokenization strategies) and show how different design choices cascade through the entire pipeline. By the end, you'll have a mental framework for evaluating not just these four models, but the next wave of TTS systems as they emerge.

This work sits within the SEI's AI Division's broader research on AI-enabled planners, and our TTS exploration grew directly out of a recurring mission partner question: how do you keep a human decision-maker in the loop when they already have their hands full?

Several concepts and open-source tools are used by the four TTS systems we discuss later in this post:

Phonemes & espeak-ng

Phonemes are the smallest units of sound that distinguish one word from another (e.g., “cat” has three: /k/, /æ/, /t/). espeak-ng is an open-source, rule-based tool that converts written text into phoneme sequences. This conversion is useful because phonemes represent how words are pronounced, bypassing tricky spelling inconsistencies (e.g., “through” versus “threw”). espeak-ng is used by NeuTTS and Piper.

Mel Spectrograms

A mel spectrogram is a two-dimensional representation of audio showing frequency content over time. It’s created by

1. Windowing—Breaking the audio into overlapping time chunks (frames), typically 20-50 milliseconds (ms) each
2. FFT—Applying a Fast Fourier Transform to each frame to extract frequency components
3. Mel scaling—Mapping frequencies to the mel scale, which matches human hearing perception (we're more sensitive to differences at low frequencies)

The result shows what sounds are present but discards phase information (the exact wave shape). Many TTS systems generate mel spectrograms as an intermediate step, then use a vocoder to convert them to audio. Mel spectrograms are used by Piper and Chatterbox.

Figure 1: Mel spectrogram of a human voice saying “Tally 2 technical, stationary. Weapons free. First Apache, action 40, guns away. Second Apache, 6 nails away.”

Neural Audio Codec

Neural codecs compress raw audio into compact token sequences using learned encoder-decoder networks. NeuTTS uses NeuCodec (dual encoders for semantic + acoustic features, FSQ quantization). VibeVoice uses a variant of a variational autoencoder, σ-VAE, which fixes the standard deviation. σ-VAE achieves 3200x compression at just 7.5 tokens/second. Neural codecs enable LLMs to “speak audio” by predicting tokens instead of raw samples.

Vocoders

Vocoders convert mel spectrograms into audio waveforms. HiFi-GAN (used by Piper and Chatterbox) upsamples using transposed convolutions to reconstruct 22kHz+ waveforms of audio from ~80 frames/sec of mel frames, and was trained adversarially to produce natural-sounding output. Neural codec decoders (NeuTTS, VibeVoice) serve a similar role.

LLM Backbones

Modern TTS increasingly uses large language model (LLM) architectures. NeuTTS fine-tunes Qwen 0.5B, VibeVoice uses Qwen2.5 (1.5B/7B), and Chatterbox uses Llama (500M). These LLMs are adapted to predict audio tokens/features instead of text tokens, leveraging their ability to model long-range dependencies. Most of these models generate audio sequentially, predicting one frame/token at a time, in an autoregressive fashion. This sequential prediction enables coherent long-form output but limits generation speed and maximum length (bounded by context window). Piper is the exception, using a non-autoregressive VITS architecture.

Criteria for Model Comparison

Before diving into each model, it helps to establish the dimensions along which we'll compare them. These criteria emerged naturally from studying the four architectures and capture the key trade-offs in TTS design:

• Architecture type — The model's backbone and how it produces audio: which LLM (if any) drives the generation, and what component decodes the model's output into a waveform (neural codec, vocoder, or direct synthesis). This tells you the model's lineage and which design family it belongs to — VITS, LLM-plus-codec, or LLM-plus-vocoder — which in turn implies its trade-offs in speed, quality, and scalability.
• Parameters — Model size, ranging from ~20M (Piper) to 1.5B (VibeVoice). Larger models generally produce more natural speech but need more compute and memory.
• Voice Cloning — Whether the model can replicate a specific voice from a short reference clip (zero-shot), or requires separately trained voice models. Also captures how much reference audio is needed (3–15s) and whether a transcript is required.
• Multi-Speaker — Whether the model can generate multiple distinct speakers within a single output, essential for conversational content like podcasts and audiobooks.
• Max Duration — The longest continuous output the model can produce. Autoregressive models are bounded by their LLM context window (~30s to 90 min); non-autoregressive models like Piper have no hard limit.
• Prosody Control — Whether the model supports mechanisms beyond plain text to influence delivery — paralinguistic tags like [laugh] and [cough], SSML markup, or punctuation-based pacing.
• Generation — Whether audio is produced in parallel (fast, one-shot) or autoregressively (sequential, slower but more coherent for long-form output).
• Output Sample Rate — The fidelity of the output waveform. Piper and Chatterbox output at 22 kHz, while NeuTTS and VibeVoice output at 24 kHz. Higher sample rates capture more frequency detail, though the perceptual difference at these rates is subtle and output quality depends far more on the model architecture than the sample rate alone.
• Phonemizer — How text is converted to the units the model processes. Piper and NeuTTS first convert text to phonemes using espeak-ng (a rule-based phonemizer) and then tokenize those phonemes — this captures pronunciation explicitly but ties the model to a specific language's phoneme set. VibeVoice and Chatterbox skip phonemization entirely, using BPE tokenizers on raw text, which is language-agnostic but leaves the model to learn pronunciation implicitly from training data.

These criteria frame the comparisons in the deep dives below and are summarized in the matrix at the end.

Model Deep Dives

NeuTTS Air

NeuTTS Air is a TTS model developed by Neuphonic that brings voice cloning capabilities to edge devices. At its core is a fine-tuned Qwen 0.5B language model, making it one of the first TTS systems to leverage a general-purpose LLM for speech synthesis.

How It Works

The model operates in two stages. First, input text is converted into phonemes using espeak-ng. These phonemes, along with acoustic tokens extracted from a reference audio clip, are fed into the fine-tuned Qwen model. The Qwen LLM has been trained in the fine tuning to predict new acoustic code tokens that represent the desired speech. Importantly, Qwen never “hears” audio directly. Instead, the reference audio is first encoded into tokens by NeuCodec, so the entire pipeline operates in a shared token space. In the second stage, these predicted acoustic tokens are decoded back into audio by NeuCodec's decoder, producing a 24kHz waveform.

Figure 2: NeuTTS Air model operation

NeuCodec: The Neural Audio Codec

NeuCodec uses a dual-encoder design where two separate encoders process the input audio in parallel:

• Wav2Vec2-BERT captures semantic and linguistic features, essentially “understanding” what is being said
• BigCodec captures acoustic features like timbre, pitch, and voice characteristics

The outputs from both encoders are combined and quantized using Finite Scalar Quantization (FSQ). Unlike traditional vector quantization which learns a codebook of embeddings, FSQ simply rounds continuous values to a fixed set of discrete levels. This avoids training instabilities like codebook collapse while achieving 50 tokens-per-second at just 0.8 kbps.

Figure 3: NeuCodec design

Voice Cloning

To clone a voice, you need a reference audio clip (3 to 15 seconds of clean speech) and a transcript of what is being said. The model uses the transcript to learn which sounds correspond to which parts of the audio, allowing it to apply those voice characteristics to new text.

Limitations

• context window of ~2048 tokens limits output to roughly 30 seconds
• no prosody control (no SSML or markup support)
• espeak-ng is hardcoded to American English phonemes
• longer content requires chunking and stitching

Piper

Piper is a fast, local neural text-to-speech engine from the Open Home Foundation. Built on the VITS architecture (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech), it's designed to run efficiently on CPUs and edge devices.

How It Works

Unlike LLM-based models, Piper is non-autoregressive, which means it generates the entire utterance in one forward pass rather than predicting tokens sequentially.

The pipeline has two main stages. First, input text is converted to phonemes using espeak-ng, then passed through a transformer encoder that produces a rich representation of each phoneme. A duration predictor determines how long each phoneme should last, and the encoder output is expanded (repeated) to match the audio’s time resolution.

In the second stage, a normalizing flow adds natural variation to the expanded representation, and a HiFi-GAN decoder upsamples it directly to a 22kHz audio waveform.

Figure 4: Piper pipeline

VITS Architecture

VITS combines several components into one end-to-end model:

• Transformer Encoder processes phoneme embeddings with bidirectional attention.
• Duration Predictor estimates frame counts per phoneme, trained using monotonic alignment search (MAS).
• Normalizing Flow learns invertible transformations that capture speaking style variation.
• HiFi-GAN Decoder uses transposed convolutions with Multi-Receptive Field Fusion to generate raw audio

During training, a posterior encoder and MAS work together to find phoneme-to-audio alignments. At inference, only the text path is used.

Voices

Each Piper voice is a separately trained Open Neural Network Exchange (ONNX) model file. Voices capture timbre, accent, pitch range, and speaking style from their training data. Switching voices means loading a different model—there is no zero-shot cloning capability.

Limitations

• no voice cloning (must train or download pre-made voices)
• limited prosody control (punctuation influences pacing)
• espeak-ng phonemization can struggle with heteronyms
• quality depends entirely on training data for each voice

VibeVoice

VibeVoice is a text-to-speech model from Microsoft designed for expressive, long-form, multi-speaker conversational audio like podcasts and audiobooks. It can generate up to 90 minutes of speech with up to 4 distinct speakers.

How It Works

VibeVoice combines three components: ultra-low frame rate speech tokenizers, an LLM backbone, and a diffusion head.

Input text (with speaker tags like “Speaker 1: ...”) is tokenized alongside voice conditioning from reference audio. The LLM (Qwen2.5, fine-tuned end-to-end) processes this context and outputs hidden states for each token position. A lightweight diffusion head then denoises these hidden states into continuous VAE latents, which are decoded into 24kHz audio.

The key insight is operating at just 7.5 tokens per second—each token represents ~133ms of audio. This means 90 minutes of speech requires only ~40,000 tokens, fitting within modern LLM context windows.

Figure 5: VibeVoice pipeline

σ-VAE: The Acoustic Tokenizer

VibeVoice uses a σ-VAE variant (from LatentLM) that achieves 3200× compression. Unlike standard VAEs where the encoder learns both mean (μ) and variance (σ), the σ-VAE encoder only learns μ. The variance is sampled from a fixed prior distribution N(0, C_σ), preventing the variance collapse that plagues standard VAEs in autoregressive settings.

The architecture uses seven stages of transformer blocks with 1d depthwise causal convolutions (~340M parameters each for encoder and decoder).

Figure 6: σ-VAE architecture

Next-Token Diffusion

Instead of predicting discrete tokens, the LLM produces continuous embeddings that a small diffusion head (~123M params, just 4 layers) refines. At inference, it uses only 10 denoising steps with DPM-Solver++ and Classifier-Free Guidance (scale 1.3). This avoids the quality loss from discretization while remaining efficient.

Model Variants

Voice Cloning

For the 1.5B and 7B models, reference audio is passed through the VAE encoder to extract voice characteristics on the fly, no pre-training on specific speakers required. The 0.5B streaming model uses pre-computed embeddings for faster inference but is limited to predefined voices.

Limitations

• maximum duration bounded by LLM context window (not architecture)
• autoregressive generation is slower than parallel methods like Piper
• requires ~10 seconds of reference audio for cloning
• no text-based voice description (must provide audio sample)
• speaker tags required in input text for multi-speaker output

Chatterbox

Chatterbox is a family of open-source TTS models from Resemble AI, built on a Llama backbone and trained on over 500,000 hours of audio. It offers zero-shot voice cloning and paralinguistic control (e.g., [laugh] and [cough] tags).

How It Works

Text is tokenized via BPE and converted to embeddings with RoPE positional encoding. Reference audio (~10 seconds) is converted to a mel spectrogram, then passed through a speaker encoder (trained with contrastive loss) to extract a voice embedding.

These two streams merge via cross-attention: text embeddings form the Query, while the speaker embedding is projected into separate Key and Value representations. The Llama backbone (500M params) then autoregressively generates mel spectrogram frames, each frame conditions on text, speaker, and all previously generated frames.

Finally, a HiFi-GAN vocoder upsamples the mel spectrogram to a 22kHz audio waveform.

Figure 7: Chatterbox pipeline

Model Variants

The Turbo variant uses a distilled decoder that generates mel spectrograms in one step instead of 10, significantly improving speed.

Voice Cloning

Provides ~10 seconds of reference audio, and Chatterbox extracts speaker characteristics via the speaker encoder. No transcript of the reference is needed (unlike NeuTTS).

Limitations

• Context window limits output to ~50 seconds (depends on mel frame rate).
• Longer content requires chunking and crossfade stitching.
• Autoregressive generation is slower than parallel methods like Piper.
• Built-in PerTh watermarking (may or may not be desirable).

Model Comparison

Voice Cloning: Listen & Compare

In this section, we demonstrate the performance of NeuTTS, VibeVoice, and Chatterbox—TTS systems with voice cloning capability—after training on a short clip of reference audio.

Choosing the Right TTS System for Your Needs

These four TTS systems represent distinct tradeoffs in the design space:

• Piper delivers unmatched speed through parallel generation but sacrifices naturalness and voice cloning.
• NeuTTS Air achieves impressive zero-shot cloning with minimal reference audio (as little as 3 seconds), leveraging an LLM backbone in a compact package.
• VibeVoice excels at long-form, multi-speaker content (up to 90 minutes), though its streaming variant trades quality for real-time performance.
• Chatterbox balances speed, quality, and expressiveness with paralinguistic control that the others lack.

In our testing, Piper and VibeVoice Streaming produced noticeably robotic output—fine for utility applications, but not for content where naturalness matters. Chatterbox achieved lightning-fast generation, solid voice cloning, and the ability to inject [laugh] or [cough] for more human-like delivery. NeuTTS Air had similar results to Chatterbox, which is particularly impressive given its small footprint and excellent cloning quality from just a few seconds of reference audio.

The right choice depends on your constraints:

• edge deployment without cloning → Piper.
• long-form podcasts → VibeVoice 1.5B/7B.
• quick, expressive cloning with personality → Chatterbox or NeuTTS.