Skip to content

Add a native Qwen3 MLX streaming adapter - #425

Open
clkao wants to merge 23 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/mlx-qwen3-asr-backend
Open

clkao wants to merge 23 commits into
QuentinFuxa:mainfrom
clkao:spacedock-ensign/mlx-qwen3-asr-backend

Conversation

@clkao

@clkao clkao commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Add an optional mlx-qwen3-asr adapter using the native incremental stream. Model loading and warmup happen once at startup; decoding runs in WLK's counted inference executor under a shared lock, with separate state and language/context options for each session.

Session language validation uses the model’s declared 30-language inventory; unsupported Whisper languages are rejected. Stable text is committed directly. Pauses, speaker changes and EOF drain finish_streaming() without a second full-utterance decode. Preserve spaces and reject revisions of an already confirmed prefix instead of duplicating text. The proposed generic ASR transform layer, timestamp wrapper and their large unit-test suite are removed. Timestamps describe chunk bounds, not word alignments.

Validation: The final head passes all required CI checks, including real-audio boundary scenarios; three adapter scenarios cover boundaries, session isolation and failed prefix revisions. A frozen extra installation on M5 completed real English/French/Chinese streams. Ruff and lock checks passed. The lock adds the candidate package without upgrading existing package versions.

The evaluated head (24082a0) includes explicit pipeline failure reporting and schema-3.1 packet pacing from #443. A real shared-model check also matches sequential and concurrent English/Chinese transcripts exactly.

The completed M5 evaluation does not support merging this head. All sixteen reports were measured on AC power (#444, benchmarks/evaluations/m5-20261107/). English/French WER is 27.65%/44.68% and Chinese CER 28.46%: 16.0/27.0/2.7 points worse than Whisper small and 6.3/20.0/16.6 points worse than Qwen Metal with the same model revision. EOF p95 is slower than both baselines in every pass, and the MLX allocator peak is higher. Most of the gap with Metal is insertions and deletions: hypotheses drop spans at segment boundaries and repeat others, which points to the adapter or incremental finalizer rather than the model.

The ten-minute Chinese stream fails reproducibly after 101.5 s: at a pause, start_silence() calls finish_streaming() with a 160-sample tail and mlx-qwen3-asr 0.3.5 raises Audio too short for log-mel extraction. The restored README, badge and published figures are unchanged.

@clkao
clkao requested a review from QuentinFuxa as a code owner August 29, 2026 19:38
@clkao
clkao marked this pull request as draft August 30, 2026 17:55
@clkao
clkao force-pushed the spacedock-ensign/mlx-qwen3-asr-backend branch 2 times, most recently from 9e3985e to f2baf2b Compare August 30, 2026 23:40
@clkao
clkao marked this pull request as ready for review August 30, 2026 23:40
@clkao
clkao force-pushed the spacedock-ensign/mlx-qwen3-asr-backend branch from f2baf2b to e94d525 Compare September 1, 2026 16:33
clkao and others added 9 commits September 1, 2026 22:14
The mlx-qwen3-asr backend (pure-MLX Qwen3-ASR, no torch/transformers/WebSocket
sidecar) built through a new generalized wrapper layer. The wrapper factors
the two jobs every non-transducer ASR duplicates: Job 1 (stable/unstable
split for models that revise) in asr_commit.py, Job 2 (timestamp manufacture
for forward-emit models) in asr_timestamps.py, composed via asr_wrapper.py.
A second provider (Voxtral-MLX) shares the timestamp module.
…guage

Finding 1 (AlignAtt seam): get_buffer returned the full rolling text\n(including the committed stable prefix), double-counting the prefix for\nconsumers that read get_buffer (display, and the AlignAtt translator\nwhich drafts over the tail via HypothesisTail). Return text[len(stable):]\n(the unstable tail only), matching WLK's contract (process_iter returns\nthe committed ASRToken list; get_buffer returns the unstable tail).

Finding 2 (per-session language): SessionASRProxy delegates __getattr__\nto the shared ASR, so getattr(asr, 'language') returned the server-wide\ndefault, ignoring the per-session ?language= override. Prefer the proxy's\n_session_language when present so a multi-language WLK server transcribes\neach session in its own language.
…n wiring

Blocker 1 — get_buffer contract conflict:
StableCommitTransform read inner.get_buffer() as the full rolling hypothesis,
but get_buffer returns only the unstable tail (WLK contract). The transform
could not compute a stable prefix and emitted garbage deltas omitting the
committed prefix. Add get_hypothesis() to MlxQwen3AsrOnlineProcessor returning
the full rolling text (Transcript with text=self._text); update
StableCommitTransform to call inner.get_hypothesis() for the full hypothesis.
get_buffer stays the tail (WLK contract, for AlignAtt/HypothesisTail). Add a
falsifiable test exercising the transform + get_hypothesis seam (fails when the
transform reads get_buffer).

Blocker 2 — out-of-scope translation wiring:
Re-carve config.py/core.py/parse_args.py to drop the translation-backend
wiring that belonged to a different PR (model field, import,
translation-backend choice, flag, qwen3+NLLB guard removal). Restore the
qwen3+NLLB guard. The PR3 diff is now ASR-only.
_finalize_utterance emitted the full re-decoded text at finalization, but
the stable prefix was already emitted during streaming by StableCommitTransform
→ text appeared twice. Fix: StableCommitTransform now tracks the cumulative
committed text on inner._emitted_stable; _finalize_utterance uses
_compute_finalize_delta to emit only the uncommitted suffix (or full text
when no streaming commit happened, or full corrected text when the re-decode
corrected the prefix). 7 new tests including an integration test that
exercises the transform + dedup path end-to-end and would fail on pre-fix
code (verified: pre-fix produces 'alpha beta alpha beta gamma delta').
… override tests

- Add [tool.uv].conflicts for mlx-qwen3-asr vs qwen3-streaming/qwen3-vllm/
  qwen3-vllm-metal (transformers>=5 vs ==4.57.6), matching the mlx-llm-mt
  pattern. Update uv.lock conflicts section to match.
- Add two-session ASR isolation regression test: two MlxQwen3AsrOnlineProcessor
  instances from shared model cache have independent per-session state
  (_state, _text, _stable_text, _emitted_stable, _utt_audio); model object
  is shared (loaded once).
- Add per-session language override regression test: SessionASRProxy with
  language='ja' causes the processor to resolve to Japanese, not the
  server-wide default.
…P fallback

The StableCommitTransform re-derived stability from the rolling hypothesis
via LCP on split_text_units, which splits on whitespace only. CJK has no
whitespace, so a 30s zh transcript collapsed into ONE unit, LCP was always 0,
and the transform never committed during streaming (zh->en: 0 provisionals,
1 final at finish).

Three changes:

1. Prefer the backend's NATIVE stable prefix (prefer_native_stable=True,
   default). Backends that expose a monotonic stable field (mlx-qwen3-asr's
   _stable_text) commit it directly — no LCP re-derivation. This is
   language-agnostic (the model's own field works on CJK and Latin alike).
   zh->en: 0 -> 11 streaming commits, first at 6s. en->zh: 7 streaming,
   first at 6s.

2. split_text_units: CJK char-split fallback. When native stable_text is
   unavailable, the LCP path now splits CJK runs into character units (one
   char = one unit ≈ one BPE token) instead of collapsing them. Detection is
   by Unicode range (not a lang switch), applying to zh/ja/ko equally.

3. Token-LCP variant (tokenize_fn): commit on the model's own token strings
   instead of text units. Normalizes CJK (1 char ≈ 1 token) and Latin
   (subword) to the same token budget. Faster first commit (4s) but fragments
   mid-word; kept as an option, not the default.

No language-specific switches. The only lang-aware code is CJK Unicode-range
detection in the fallback char-split.
@clkao
clkao force-pushed the spacedock-ensign/mlx-qwen3-asr-backend branch from e94d525 to 7610b2a Compare September 2, 2026 05:16
@QuentinFuxa QuentinFuxa changed the title feat: mlx-qwen3-asr ASR backend + generalized ASR wrapper layer Add a native Qwen3 MLX streaming adapter Sep 7, 2026
QuentinFuxa added a commit that referenced this pull request Nov 8, 2026
All sixteen reports of the M5 screening, measured in one serial session on AC
power with the same commits, models and corpus as the partial September series.
The September directory is kept unchanged.

Native Qwen (#425) fails every comparison: its WER/CER is worse than both
Whisper small and Qwen Metal, EOF is slower, and the ten-minute stream fails
reproducibly on a 160-sample tail at a pause. Nemotron (#426) passes French
against both baselines but fails English against Whisper and Chinese against
Metal on quality.
@QuentinFuxa

Copy link
Copy Markdown
Owner

Sorry for the long silence on this one. I measured it against the existing backends on an M5 (the full series is in #444) and ran it on real audio, and it is not mergeable yet. The integration itself is clean: the adapter is small, the only conflict with main is uv.lock, and the tests pass. The problem is the output.

Measurements (#444, 90 FLEURS clips per language, three passes)

Native Qwen (this PR) Qwen Metal, same weights Whisper small MLX
English WER 27.7% 21.4% 11.7%
French WER 44.7% 24.7% 17.7%
Chinese CER 28.5% 11.8% 25.8%

EOF p95 is also slower than both baselines (0.9 to 1.9 s). Since Metal uses the same model revision, the gap comes from the adapter or the incremental finalizer, not the model. In French it is mostly insertions and deletions: 441 and 330, against 81 and 84 for Metal.

What I see on real audio

On audio_tests/00_00_16_french_1_speaker.wav, "là il y a eu tel second de silence et je" comes out twice in a row. On 00_00_30_english_3_speakers.wav, "It does a pretty good job filtering noise." is repeated, the next sentence is cut and the end of the file is missing.

I traced the repetition to finish_streaming() in mlx-qwen3-asr 0.3.5. The state text is correct before the call and doubled after it. When the tail decode adds nothing, the library re-transcribes the last window (last chunk plus pending tail) and appends the result to the existing text, so the same words appear twice. That refinement call also does not receive the session language. The adapter calls finish_streaming() at every pause and speaker change, so this happens at every segment boundary.

Two other things:

  • The ten-minute Chinese stream fails after 101.5 s, reproducibly. At a pause, start_silence() calls finish_streaming() with a 160-sample tail and the library raises Audio too short for log-mel extraction.
  • In my trace stable_text stays empty for the whole utterance, so nothing is committed before the pause. With 2 s chunks and the first two chunks kept unstable, short utterances never reach a stable prefix. That explains the latency numbers.

What would make it mergeable

  1. No repeated text at boundaries: either disable the tail refinement (enable_tail_refine) or deduplicate in the adapter, with a test that goes through AudioProcessor on a recording with pauses.
  2. No exception on a short tail at a pause.
  3. A rerun of the three FLEURS languages showing it is at least on par with Qwen Metal, since the weights are the same.

The first one may be better fixed upstream in mlx-qwen3-asr. If you don't have time for this now, no problem, tell me and I'll mark the PR as on hold rather than leave it open without an answer.

@clkao
clkao force-pushed the spacedock-ensign/mlx-qwen3-asr-backend branch 2 times, most recently from ef7a966 to bc9e18f Compare November 14, 2026 02:07
… 0.4.4

0.4.x replaced the finish-time tail refinement pass with a window re-decode that includes the
pending tail, so a segment's text is produced once and the adapter commits only what clients do
not already hold. Adds an agreed-prefix fallback while stable_text is empty, repairs a non-prefix
or rewriting candidate into an append (upstream QuentinFuxa#31) instead of aborting, and ends a segment whose
leftover audio is below the log-mel floor instead of raising.
@clkao
clkao force-pushed the spacedock-ensign/mlx-qwen3-asr-backend branch 3 times, most recently from bc9e18f to d538a8e Compare November 14, 2026 02:10
@clkao

clkao commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

d538a8e fixes all three. It sits on top of this branch, so no existing commit was rewritten, and I
merged main so the uv.lock conflict is gone.

  • Duplication at boundaries. The tail refinement pass you traced is gone in 0.4.2 and later
    (PR #26). This branch requires 0.4.4, and the
    adapter commits only the text that clients do not already hold. test_pause_in_audio_commits_each_sentence_once
    drives AudioProcessor over audio with pauses.
  • Short tail at a pause. Version 0.4.4 still rejects input below 200 samples, so the adapter
    keeps the decode window at or above 400 samples and ends the segment instead of raising. The
    600 s Chinese stream from #444 now runs to the end: 1 of 1 ok, 240 tokens, 8 lines, EOF 0.001 s.
  • Quality, same weights. Model pinned to snapshot_revision 5eb1441… with a sha256 for each
    file. Corpus corpus_sha256 95586eb7…. Command: wlk bench --warmup --speed 1, 30 clips per
    language.
this branch qwen-native before Qwen Metal Whisper small
zh CER 11.15% 28.46% 11.85% 25.77%
en WER 13.89% 27.65% 21.36% 11.66%
fr WER 14.39% 44.68% 24.68% 17.66%

Likely cause of the gain: this backend commits text after a re-decode of a window of up to 30 s
that includes the pending tail. The Metal path commits with 0.64 s of right context
(qwen3_streaming_right_context_ms), and that horizon falls inside a word in English and French,
but rarely in Chinese. Two facts fit that: the two systems give byte-identical hypotheses on
16 of 30 Chinese clips and 3 of 30 English clips, and substitutions carry the whole gap while
insertions are equal.

@QuentinFuxa

Copy link
Copy Markdown
Owner

Big improvement, thanks. I checked it on the same manifest (one paced pass, --speed 1, French and English): English WER is 13.89%, exactly your figure, and French is 14.97% (you have 14.39%, close enough for a single pass). EOF p95 is 0.19 s, all 60 clips ok, valid timestamps. The ten-minute crash and the English repetitions are gone on my recordings too.

One bug is left before I can merge: revisions near the start of a segment still duplicate text. On audio_tests/00_00_16_french_1_speaker.wav the output reads "là il y a eu telLà il y a eu tel second de silence". In the FLEURS run, 7 of the 30 French clips have the same pattern, for example fleurs_fr_1935: "cependant ce n'est pas vrai bienCependant ce n'est pas vrai bien qu'il y ait…", and fleurs_fr_1975: "c'est le plusC'est le plus petit pays…".

The trace on the French recording:

source_before = 'là il y a eu tel'
candidate     = 'Là il y a eu tel second de silence et je repars'
delta         = 'Là il y a eu tel second de silence et je repars'

The model only capitalized the first word. _append_only_delta sees that the candidate does not start with the source and that it is longer, so it takes the part after the common prefix. The common prefix is empty because of the capital L, so the whole segment is appended again, and with no separator. Any revision of an early word does the same.

Since clients cannot take back text, I'd suggest appending only what goes beyond what they already hold: when the candidate rewrites the source and is at least as long, emit candidate[len(source):] (adjusted to a word boundary) and accept that the earlier wording stays. A case- and whitespace-insensitive comparison would already catch the capitalization cases. A regression test with exactly this trace would pin it.

With that fixed I'm happy to merge: French beats both baselines, Chinese matches Metal by your numbers, and English is close to Whisper small.

A window re-decode can capitalise the first word of the same sentence. The shared-prefix
comparison was case-sensitive, so a candidate that only changed 'là' to 'Là' reported no shared
prefix, and the whole segment reached clients a second time ('là il y a eu telLà il y a eu tel
second de silence').

Comparing with casefold makes that candidate extend the committed text, so only the tail is sent.
Slice at the shared prefix rather than at len(source): slicing at len(source) drops the first
letters of a word when a rewrite cuts inside one, which broke the upstream QuentinFuxa#31 cases.

Tests replay the reported trace and the fr_1935 form of it, and assert that what clients hold plus
the delta reconstructs the candidate when a rewrite cuts inside a word.
@clkao

clkao commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in db10f6c. Your trace now yields delta = ' second de silence et je repars'.

The cause was the case-sensitive shared-prefix comparison. Your candidate only capitalises the
first word, so the shared prefix measured zero and the segment went out again. Comparing with
casefold makes it a candidate that extends the committed text.

One deviation from your suggestion: I did not slice at len(source). When a rewrite cuts inside a
word, slicing at len(source) drops the first letters of that word, and it broke the four upstream
#31 cases in the test file. Keeping the slice at the shared prefix, with a case-insensitive
comparison, fixes your trace and preserves those. For the cut-inside-a-word case the test asserts
that what clients hold plus the delta reconstructs the candidate, so nothing is sent twice and no
word is lost.

Tests replay your exact trace, the fleurs_fr_1935 form of it, and that cut-inside-a-word case.

What I could not do: reproduce the 7 of 30 on French clips. audio_tests/ is not in the repo, and
in my FLEURS runs no French hypothesis repeats its opening span once legitimate repeats are
excluded ("ce n'est pas vrai … ce n'est pas une carte"). French weighted WER moved from 14.39% to
13.92% with the fix, so it did change output. Please re-run your check on
00_00_16_french_1_speaker.wav and on those French clips.

Both PRs add a backend next to the same registries, so each conflict keeps both sides:
pyproject extras and [tool.uv] conflicts now list mlx-qwen3-asr and nemotron-mlx-asr,
benchmark/compat.py maps both, parse_args.py offers both in --backend, and
backend_support.py exports both availability helpers. uv.lock regenerated with
UV_NO_CONFIG=1 (408 packages).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants