Skip to content

Add a Nemotron MLX adapter using upstream streaming caches - #426

Merged
QuentinFuxa merged 10 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/nemotron-mlx-asr-backend-rebased
Nov 14, 2026
Merged

QuentinFuxa merged 10 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/nemotron-mlx-asr-backend-rebased

Conversation

@clkao

@clkao clkao commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

Add an optional Nemotron MLX adapter using mlx-audio's incremental mel frontend and cached Conformer encoder, with a small stateful RNNT decoder loop. WLK owns voice activity detection and stream boundaries; the copied encoder, second VAD and full-utterance second pass are removed.

Use the converted mlx-community/nemotron-3.5-asr-streaming-0.6b checkpoint and mlx-audio>=0.5.1,<0.6. The previous default points to a Transformers checkpoint and fails at startup with Model type nemotron3_5_asr not supported for stt. Sessions have independent caches and language prompts, while model loading and decoding stay on one dedicated thread. A shared mutex alone was insufficient: repeated sessions failed with There is no Stream(gpu, 2) in current thread. EOF flushes the final frontend/encoder frames once. Token times are encoder emission times, not forced word alignments.

Validation: The final head passes all required CI checks, including real-audio boundary scenarios; two MLX CPU scenarios cover partial-frame draining, pauses, EOF, interleaved sessions, thread ownership and language/context handling. The frozen extra installs mlx-audio 0.5.1 and completes real English/French/Chinese streams on M5. Ruff and lock checks passed; existing Metal, Torch and Transformers versions remain unchanged.

The completed M5 evaluation does not meet the agreed rule. All sixteen reports were measured on AC power (#444, benchmarks/evaluations/m5-20261107/). Head 1842f4c completes all 270 short-clip sessions and the ten-minute Chinese stream, with valid line timestamps and quality identical to September. EOF p95 is 71% to 94% lower than Whisper small and Qwen Metal in every pass. French passes against both baselines, but English WER is 19.27% versus 11.66% for Whisper small and Chinese CER is 22.62% versus 11.85% for Qwen Metal, beyond the one-point limit. The ten-minute stream has no repetition loop; its output contains one run of twenty Thai characters.

This PR remains stacked on #425; the research translation changes are already on main. The restored README, badge and published figures are unchanged.

@clkao
clkao requested a review from QuentinFuxa as a code owner August 30, 2026 05:25
@clkao
clkao marked this pull request as draft August 30, 2026 17:55
@clkao
clkao force-pushed the spacedock-ensign/nemotron-mlx-asr-backend-rebased branch from 2671610 to 04cbe9c Compare August 30, 2026 23:34
@clkao
clkao marked this pull request as ready for review August 30, 2026 23:40
@clkao
clkao force-pushed the spacedock-ensign/nemotron-mlx-asr-backend-rebased branch 2 times, most recently from 0126dc9 to d7c1e6d Compare September 2, 2026 05:18
@QuentinFuxa QuentinFuxa changed the title feat: nemotron-mlx ASR transducer backend (depends on #423) Add a Nemotron MLX adapter using upstream streaming caches Sep 7, 2026
QuentinFuxa added a commit that referenced this pull request Nov 8, 2026
All sixteen reports of the M5 screening, measured in one serial session on AC
power with the same commits, models and corpus as the partial September series.
The September directory is kept unchanged.

Native Qwen (#425) fails every comparison: its WER/CER is worse than both
Whisper small and Qwen Metal, EOF is slower, and the ten-minute stream fails
reproducibly on a 160-sample tail at a pause. Nemotron (#426) passes French
against both baselines but fails English against Whisper and Chinese against
Metal on quality.
@QuentinFuxa

Copy link
Copy Markdown
Owner

I measured this one together with #425 (series in benchmarks/evaluations/m5-20261107/) and ran it on real audio. It works well: correct text on the English and French recordings in audio_tests/, EOF delay between 0.02 and 0.17 s, valid timestamps, and three concurrent sessions run without the MLX thread error. In the FLEURS series it beats Whisper small MLX in French (15.0% vs 17.7% WER) and Chinese (22.6% vs 25.8% CER), with EOF p95 around 0.1 s.

I'd like to merge it. What it needs:

  1. Make it independent of Add a native Qwen3 MLX streaming adapter #425. It currently includes the Qwen adapter, which is not ready (see my comment there). Could you rebase it alone on main, with only the Nemotron files?
  2. Regenerate uv.lock on current main, it is the only conflict.
  3. The output has double spaces between sentences (test. We want). Probably the token text joined with a leading space after punctuation.
  4. English WER is 19.3% against 11.7% for Whisper small, so docs/backends.md should say it is a low-latency option that is strongest in French and Chinese rather than a general replacement.

One note for later: #450 pins mlx-audio<0.5 and this needs >=0.5.1, so the two cannot be installed together yet.

Pin mlx-audio>=0.5.1,<0.6 for Apple Silicon and declare it conflicting
with qwen3-streaming, qwen3-vllm and qwen3-vllm-metal: those pin
transformers==4.57.6 while mlx-audio requires transformers>=5.

uv.lock is regenerated from origin/main's lock under UV_NO_CONFIG=1. The
delta adds mlx-audio, miniaudio, httpx2-jsfetch and win32-setctime, and
changes no existing version; the nvidia, cuda-toolkit and torch sets are
unchanged. Regenerate from main's lock, not from a branch lock: the
latter carries fork state forward and invents a CUDA downgrade.
Streaming FastConformer-RNNT through mlx-audio's incremental mel frontend
and cached Conformer encoder, with a small stateful RNNT decode loop that
keeps decoder state between pushes.

WLK owns voice activity detection and stream boundaries. Pauses, speaker
changes and EOF flush the final mel/encoder frames once; there is no
copied encoder, no second VAD and no full-utterance re-decode. Sessions
get independent caches and language prompts. Model loading and decoding
stay on one dedicated thread because cached MLX streams cannot migrate
between caller threads; a shared mutex alone produced
'There is no Stream(gpu, 2) in current thread'.

Token times are encoder emission times clipped to received audio, not
forced word alignments. Session terminology context is rejected through
the non-prompt backend path.
Position it as a low-latency option rather than a general replacement,
with the measured figures: French is better than both baselines, Chinese
beats Whisper small but trails Qwen Metal, and English is 7.6 points
behind Whisper small.
The model closes an utterance with a standalone '▁' token and opens the
next one with a space-prefixed first word, so both encode the same
separator. WLK joins committed pieces verbatim, which reads
'document.  It measures'. Drop the whitespace-only token that ends a
flushed utterance (and trim a trailing space from its last token); the
next utterance's leading space is kept because it is the separator.

Reproduced end-to-end on the FLEURS clips from the M5 evaluation
(fleurs_en_1682/1700/1711), which produce the same hypothesis text and
WERs as the recorded reports: 4 double spaces before, 0 after, WER
unchanged at 39.39 / 0.00 / 8.82. Interleaved with the pre-fix code the
ASR RTF is unchanged (0.277 before, 0.243 after).
…d registration

docs/backends.md compressed the review's two latency ranges into one
('71% to 94% below Whisper small and Qwen Metal'). Restore the per-baseline
figures the review states (71% to 87% below Whisper small, 74% to 94% below
Qwen Metal).

parse_args.py had lost the newline between the --backend choices list and
its help string when the backend was added, and backend_support.py gained a
third blank line between functions.
Four frames through process_iter() with no finish() call: the RNNT decode
must already carry non-None, non-decreasing, distinct per-frame ASRToken
start/end times. Mutating the decode to drop, collapse or reverse the times
fails the test.
The decoder emits a standalone space token and then a space-prefixed word
whenever a word starts after a token boundary, so joined text reads
'test.  We'. Collapse each run to one space, carrying the state of the
last returned text across decode calls and across a finish that emits no
token. Collapsing to zero would glue two utterances into 'the.We', which
is why exactly one space survives.
…ream artifact

Moves the section into the backend list so Environment constraints stays last,
drops the candidate qualifier, and states the ten-minute stream's one artifact
instead of claiming a clean run.

Also puts the two config fields after session_idle_timeout, where the file
keeps new fields.
…e set

A final chunk whose only token is the closing space leaves the flag set by that
token, which is then trimmed away without ever reaching the caller; the next
utterance dropped its leading space and the two lines glued into 'WeWe.'.
Restore the flag the previous call left whenever a final chunk returns nothing.
@clkao
clkao force-pushed the spacedock-ensign/nemotron-mlx-asr-backend-rebased branch from 1842f4c to edc5860 Compare November 14, 2026 01:53
@clkao

clkao commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

The double space is fixed. The decoder emits a space token, then a word token that starts with a space. The adapter now collapses each space run to one. In 60 cached FLEURS clips in English and Chinese, there are zero double spaces and the word error rate is the same.

docs/backends.md uses the m5 figures. French is better than both baselines. Chinese beats Whisper small. English is 7.6 points behind Whisper small.

@QuentinFuxa
QuentinFuxa merged commit 496b5f6 into QuentinFuxa:main Nov 14, 2026
9 checks passed
@QuentinFuxa

Copy link
Copy Markdown
Owner

Merged, thanks. The space fix and the docs read well, and with the branch independent of #425 it went in cleanly. I re-ran the English and French recordings on the final branch: correct text, no double spaces, three concurrent sessions fine.

clkao added a commit to clkao/WhisperLiveKit that referenced this pull request Nov 14, 2026
Both PRs add a backend next to the same registries, so each conflict keeps both sides:
pyproject extras and [tool.uv] conflicts now list mlx-qwen3-asr and nemotron-mlx-asr,
benchmark/compat.py maps both, parse_args.py offers both in --backend, and
backend_support.py exports both availability helpers. uv.lock regenerated with
UV_NO_CONFIG=1 (408 packages).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants