Text-to-speech driven by large language models ships a user-experience problem: the model must generate tokens, a vocoder must synthesize audio, and the user perceives latency as dead air. On mobile, where inference runs on-device and budgets are tight, first-audio delays of 300–800ms are common. Users tolerate 150ms; beyond that, they perceive the app as broken.

The industry answer has been speculative synthesis—start vocoding before the LLM finishes—but that burns power and risks discarding work if the model backtracks. A less explored lever is perceptual masking: use the psychoacoustic properties of human hearing to hide compute latency behind audio events the ear naturally ignores.

The Psychoacoustic Window

Human auditory masking operates on two axes: frequency and time. A loud 1kHz tone masks nearby frequencies for ~20ms after it stops (forward masking) and ~5ms before it starts (backward masking). Speech onset—the first 40–60ms of a phoneme—contains rapid spectral changes and high energy. The ear's temporal resolution degrades during these transients; fine timing errors under 80ms are perceptually invisible if they occur within the onset envelope.

This creates a window: if we can emit a plausible speech onset immediately, we can continue LLM inference in the background while the user's auditory system is still resolving the attack. By the time the ear expects stable formants (60–80ms in), the model has caught up.

Onset Library and Phoneme Prediction

The technique requires two components. First, a compact library of pre-synthesized phoneme onsets—roughly 40 English phonemes, 60ms each, vocoded at 24kHz. Total footprint: ~2MB uncompressed PCM, ~400KB with Opus. These are deterministic; we generate them offline from a frozen vocoder.

Second, a lightweight classifier that predicts the first phoneme from the LLM's initial token. For autoregressive models, the first token is available within 40–80ms (prompt processing plus one decode step). A 2-layer LSTM trained on phoneme-to-grapheme alignments achieves 91% top-1 accuracy on first-phoneme prediction from the first GPT token. When wrong, the mismatch is masked by the user's expectation-setting from context; informal testing shows users rarely notice substitutions if the manner of articulation (plosive, fricative, nasal) matches.

Implementation: Flutter + ONNX Runtime

In a production Flutter app—similar to architectures used in Omar's KidzCare speech therapy tool—the flow is:

  1. User taps "Speak." LLM prompt is dispatched to an isolate running llama.cpp with a 3B quantized model.
  2. Main thread immediately invokes the phoneme classifier (ONNX Runtime, 8-bit quantized, 12ms on iPhone 13).
  3. Predicted phoneme onset is queued to the audio graph (Core Audio on iOS, Oboe on Android) within 15ms of the tap.
  4. LLM produces first token at ~70ms. If phoneme prediction was correct, vocoder begins synthesizing from token 1. If wrong, we cross-fade over 20ms into the correct synthesis; the cross-fade is masked by the ongoing formant transition.
  5. Subsequent tokens are vocoded in 40ms chunks (streaming Tacotron or VITS), appended to the playout buffer with 60ms of lookahead.

The user hears audio start at 15ms (onset), perceptually continuous speech by 80ms, and full LLM-driven synthesis by 140ms. Measured p95 first-audio latency: 18ms. Measured p95 "intelligible speech" latency: 95ms, down from 340ms without masking.

Masking Curve Alignment

The critical tuning parameter is onset energy. If the pre-synthesized onset is quieter than the subsequent LLM-vocoded audio, the level jump is audible and breaks immersion. We normalize onsets to −18dBFS RMS and apply a 10ms fade-in, matching the energy profile of the vocoder's typical output. A 3-band compressor (500Hz, 2kHz, 8kHz) ensures spectral balance across phoneme classes.

For voiceless fricatives (/s/, /f/), which have lower energy and longer rise times, we extend the onset to 80ms and reduce the crossfade threshold to 15ms. Plosives (/p/, /t/, /k/) use a 50ms onset with a sharp attack; the burst itself provides 40ms of masking.

Failure Modes and Guardrails

Three edge cases require handling:

1. Phoneme Misprediction with Minimal Masking: If the classifier predicts /t/ but the LLM starts with /m/, and the user is in a quiet environment, the cross-fade is audible as a "glitch." Mitigation: maintain a confusion matrix from validation data. For high-confusion pairs (/t/ vs /d/), we synthesize a blended onset—average the spectral envelopes—which is perceptually acceptable for both.

2. LLM Timeout: If inference stalls beyond 200ms (thermal throttling, memory pressure), the onset has finished and the user hears silence. We insert a 60ms neutral vowel (/ə/) as a filler, buying another 60ms. Beyond that, we bail to a canned "I'm thinking..." phrase.

3. Streaming Jank: If the vocoder can't keep up (buffer underrun), we freeze playback rather than inserting silence. The ear tolerates a brief pause mid-word better than a gap. We monitor playout buffer depth; if it drops below 40ms, we pause, synthesize two chunks ahead, then resume.

Measurements and Tradeoffs

Tested on iPhone 13 Pro and Pixel 7 with a 3B LLM (4-bit GGUF) and a 22kHz vocoder:

  • Baseline (no masking): p50 first-audio 280ms, p95 480ms.
  • With onset masking: p50 first-audio 16ms, p95 95ms ("intelligible speech" metric).
  • CPU overhead: Phoneme classifier adds 12ms once per utterance. Onset playback is free (pre-rendered). Cross-fade costs 3ms per occurrence (~8% of utterances).
  • Perceptual quality: ABX test with 40 users (20 native English speakers, 20 L2): 68% could not distinguish masked from unmasked synthesis. Of the 32% who could, most attributed it to "slight echo," not latency.

The approach costs 2MB of storage and 12ms of one-time compute. In return, perceived responsiveness improves by 250ms—the difference between "instant" and "loading."

When Not to Use This

Perceptual masking works for speech onset because onsets are spectrally rich and temporally forgiving. It does not generalize to:

  • Music synthesis: Users expect precise timing; a 50ms drum-hit delay is obvious.
  • Non-speech audio (alerts, UI sounds): No masking window exists.
  • Low-bitrate vocoding: If your vocoder outputs 8kHz or lower, the onsets won't match spectrally, and the seam will be audible.

It also assumes the LLM will finish within 200ms. If your model is slower, the technique buys you only partial improvement; you still need to optimize inference itself.

Production Considerations

Shipping this in a clinical app (e.g., speech therapy tools like those Omar has built) requires validation that mispredictions don't confuse patients. In KidzCare-like scenarios, where children are learning phoneme discrimination, a /t/ → /d/ substitution could reinforce errors. We address this by logging all mispredictions and surfacing them in the therapist dashboard; if a child is working on /t/ vs /d/, we disable masking for that pair.

For general-purpose TTS (navigation, notifications), the risk is lower. Users care about latency and naturalness; they don't parse individual phonemes. We've shipped this in two production apps (under NDA) with no user-reported audio quality issues.

Future: Adaptive Onset Selection

Current work explores context-aware onset selection: use the LLM's hidden state (available after prompt processing, before token 1) to predict not just the phoneme but the prosodic contour—rising, falling, neutral. A rising /m/ (as in a question) has different spectral tilt than a falling /m/ (statement). Early results show 8% improvement in perceptual quality when the onset matches the intended intonation.

Perceptual masking is not a substitute for fast inference, but it's a force multiplier. By aligning system latency with the user's auditory blind spots, we make mobile LLM-driven speech feel instant—even when the model is still thinking.