Repository navigation
Add a Nemotron MLX adapter using upstream streaming caches - #426
QuentinFuxa merged 10 commits into
Conversation
2671610 to
04cbe9c
Compare
0126dc9 to
d7c1e6d
Compare
All sixteen reports of the M5 screening, measured in one serial session on AC power with the same commits, models and corpus as the partial September series. The September directory is kept unchanged. Native Qwen (#425) fails every comparison: its WER/CER is worse than both Whisper small and Qwen Metal, EOF is slower, and the ten-minute stream fails reproducibly on a 160-sample tail at a pause. Nemotron (#426) passes French against both baselines but fails English against Whisper and Chinese against Metal on quality.
|
I measured this one together with #425 (series in I'd like to merge it. What it needs:
One note for later: #450 pins |
Pin mlx-audio>=0.5.1,<0.6 for Apple Silicon and declare it conflicting with qwen3-streaming, qwen3-vllm and qwen3-vllm-metal: those pin transformers==4.57.6 while mlx-audio requires transformers>=5. uv.lock is regenerated from origin/main's lock under UV_NO_CONFIG=1. The delta adds mlx-audio, miniaudio, httpx2-jsfetch and win32-setctime, and changes no existing version; the nvidia, cuda-toolkit and torch sets are unchanged. Regenerate from main's lock, not from a branch lock: the latter carries fork state forward and invents a CUDA downgrade.
Streaming FastConformer-RNNT through mlx-audio's incremental mel frontend and cached Conformer encoder, with a small stateful RNNT decode loop that keeps decoder state between pushes. WLK owns voice activity detection and stream boundaries. Pauses, speaker changes and EOF flush the final mel/encoder frames once; there is no copied encoder, no second VAD and no full-utterance re-decode. Sessions get independent caches and language prompts. Model loading and decoding stay on one dedicated thread because cached MLX streams cannot migrate between caller threads; a shared mutex alone produced 'There is no Stream(gpu, 2) in current thread'. Token times are encoder emission times clipped to received audio, not forced word alignments. Session terminology context is rejected through the non-prompt backend path.
Position it as a low-latency option rather than a general replacement, with the measured figures: French is better than both baselines, Chinese beats Whisper small but trails Qwen Metal, and English is 7.6 points behind Whisper small.
The model closes an utterance with a standalone '▁' token and opens the next one with a space-prefixed first word, so both encode the same separator. WLK joins committed pieces verbatim, which reads 'document. It measures'. Drop the whitespace-only token that ends a flushed utterance (and trim a trailing space from its last token); the next utterance's leading space is kept because it is the separator. Reproduced end-to-end on the FLEURS clips from the M5 evaluation (fleurs_en_1682/1700/1711), which produce the same hypothesis text and WERs as the recorded reports: 4 double spaces before, 0 after, WER unchanged at 39.39 / 0.00 / 8.82. Interleaved with the pre-fix code the ASR RTF is unchanged (0.277 before, 0.243 after).
…d registration
docs/backends.md compressed the review's two latency ranges into one
('71% to 94% below Whisper small and Qwen Metal'). Restore the per-baseline
figures the review states (71% to 87% below Whisper small, 74% to 94% below
Qwen Metal).
parse_args.py had lost the newline between the --backend choices list and
its help string when the backend was added, and backend_support.py gained a
third blank line between functions.
Four frames through process_iter() with no finish() call: the RNNT decode must already carry non-None, non-decreasing, distinct per-frame ASRToken start/end times. Mutating the decode to drop, collapse or reverse the times fails the test.
The decoder emits a standalone space token and then a space-prefixed word whenever a word starts after a token boundary, so joined text reads 'test. We'. Collapse each run to one space, carrying the state of the last returned text across decode calls and across a finish that emits no token. Collapsing to zero would glue two utterances into 'the.We', which is why exactly one space survives.
…ream artifact Moves the section into the backend list so Environment constraints stays last, drops the candidate qualifier, and states the ten-minute stream's one artifact instead of claiming a clean run. Also puts the two config fields after session_idle_timeout, where the file keeps new fields.
…e set A final chunk whose only token is the closing space leaves the flag set by that token, which is then trimmed away without ever reaching the caller; the next utterance dropped its leading space and the two lines glued into 'WeWe.'. Restore the flag the previous call left whenever a final chunk returns nothing.
1842f4c to
edc5860
Compare
|
The double space is fixed. The decoder emits a space token, then a word token that starts with a space. The adapter now collapses each space run to one. In 60 cached FLEURS clips in English and Chinese, there are zero double spaces and the word error rate is the same.
|
|
Merged, thanks. The space fix and the docs read well, and with the branch independent of #425 it went in cleanly. I re-ran the English and French recordings on the final branch: correct text, no double spaces, three concurrent sessions fine. |
Both PRs add a backend next to the same registries, so each conflict keeps both sides: pyproject extras and [tool.uv] conflicts now list mlx-qwen3-asr and nemotron-mlx-asr, benchmark/compat.py maps both, parse_args.py offers both in --backend, and backend_support.py exports both availability helpers. uv.lock regenerated with UV_NO_CONFIG=1 (408 packages).
Add an optional Nemotron MLX adapter using mlx-audio's incremental mel frontend and cached Conformer encoder, with a small stateful RNNT decoder loop. WLK owns voice activity detection and stream boundaries; the copied encoder, second VAD and full-utterance second pass are removed.
Use the converted
mlx-community/nemotron-3.5-asr-streaming-0.6bcheckpoint andmlx-audio>=0.5.1,<0.6. The previous default points to a Transformers checkpoint and fails at startup withModel type nemotron3_5_asr not supported for stt. Sessions have independent caches and language prompts, while model loading and decoding stay on one dedicated thread. A shared mutex alone was insufficient: repeated sessions failed withThere is no Stream(gpu, 2) in current thread. EOF flushes the final frontend/encoder frames once. Token times are encoder emission times, not forced word alignments.Validation: The final head passes all required CI checks, including real-audio boundary scenarios; two MLX CPU scenarios cover partial-frame draining, pauses, EOF, interleaved sessions, thread ownership and language/context handling. The frozen extra installs mlx-audio 0.5.1 and completes real English/French/Chinese streams on M5. Ruff and lock checks passed; existing Metal, Torch and Transformers versions remain unchanged.
The completed M5 evaluation does not meet the agreed rule. All sixteen reports were measured on AC power (#444,
benchmarks/evaluations/m5-20261107/). Head1842f4ccompletes all 270 short-clip sessions and the ten-minute Chinese stream, with valid line timestamps and quality identical to September. EOF p95 is 71% to 94% lower than Whisper small and Qwen Metal in every pass. French passes against both baselines, but English WER is 19.27% versus 11.66% for Whisper small and Chinese CER is 22.62% versus 11.85% for Qwen Metal, beyond the one-point limit. The ten-minute stream has no repetition loop; its output contains one run of twenty Thai characters.This PR remains stacked on #425; the research translation changes are already on main. The restored README, badge and published figures are unchanged.