Repository navigation
Conversation
# Conflicts: # whisperlivekit/core.py
9e3985e to
f2baf2b
Compare
f2baf2b to
e94d525
Compare
The mlx-qwen3-asr backend (pure-MLX Qwen3-ASR, no torch/transformers/WebSocket sidecar) built through a new generalized wrapper layer. The wrapper factors the two jobs every non-transducer ASR duplicates: Job 1 (stable/unstable split for models that revise) in asr_commit.py, Job 2 (timestamp manufacture for forward-emit models) in asr_timestamps.py, composed via asr_wrapper.py. A second provider (Voxtral-MLX) shares the timestamp module.
…guage Finding 1 (AlignAtt seam): get_buffer returned the full rolling text\n(including the committed stable prefix), double-counting the prefix for\nconsumers that read get_buffer (display, and the AlignAtt translator\nwhich drafts over the tail via HypothesisTail). Return text[len(stable):]\n(the unstable tail only), matching WLK's contract (process_iter returns\nthe committed ASRToken list; get_buffer returns the unstable tail). Finding 2 (per-session language): SessionASRProxy delegates __getattr__\nto the shared ASR, so getattr(asr, 'language') returned the server-wide\ndefault, ignoring the per-session ?language= override. Prefer the proxy's\n_session_language when present so a multi-language WLK server transcribes\neach session in its own language.
…n wiring Blocker 1 — get_buffer contract conflict: StableCommitTransform read inner.get_buffer() as the full rolling hypothesis, but get_buffer returns only the unstable tail (WLK contract). The transform could not compute a stable prefix and emitted garbage deltas omitting the committed prefix. Add get_hypothesis() to MlxQwen3AsrOnlineProcessor returning the full rolling text (Transcript with text=self._text); update StableCommitTransform to call inner.get_hypothesis() for the full hypothesis. get_buffer stays the tail (WLK contract, for AlignAtt/HypothesisTail). Add a falsifiable test exercising the transform + get_hypothesis seam (fails when the transform reads get_buffer). Blocker 2 — out-of-scope translation wiring: Re-carve config.py/core.py/parse_args.py to drop the translation-backend wiring that belonged to a different PR (model field, import, translation-backend choice, flag, qwen3+NLLB guard removal). Restore the qwen3+NLLB guard. The PR3 diff is now ASR-only.
_finalize_utterance emitted the full re-decoded text at finalization, but the stable prefix was already emitted during streaming by StableCommitTransform → text appeared twice. Fix: StableCommitTransform now tracks the cumulative committed text on inner._emitted_stable; _finalize_utterance uses _compute_finalize_delta to emit only the uncommitted suffix (or full text when no streaming commit happened, or full corrected text when the re-decode corrected the prefix). 7 new tests including an integration test that exercises the transform + dedup path end-to-end and would fail on pre-fix code (verified: pre-fix produces 'alpha beta alpha beta gamma delta').
… override tests - Add [tool.uv].conflicts for mlx-qwen3-asr vs qwen3-streaming/qwen3-vllm/ qwen3-vllm-metal (transformers>=5 vs ==4.57.6), matching the mlx-llm-mt pattern. Update uv.lock conflicts section to match. - Add two-session ASR isolation regression test: two MlxQwen3AsrOnlineProcessor instances from shared model cache have independent per-session state (_state, _text, _stable_text, _emitted_stable, _utt_audio); model object is shared (loaded once). - Add per-session language override regression test: SessionASRProxy with language='ja' causes the processor to resolve to Japanese, not the server-wide default.
…P fallback The StableCommitTransform re-derived stability from the rolling hypothesis via LCP on split_text_units, which splits on whitespace only. CJK has no whitespace, so a 30s zh transcript collapsed into ONE unit, LCP was always 0, and the transform never committed during streaming (zh->en: 0 provisionals, 1 final at finish). Three changes: 1. Prefer the backend's NATIVE stable prefix (prefer_native_stable=True, default). Backends that expose a monotonic stable field (mlx-qwen3-asr's _stable_text) commit it directly — no LCP re-derivation. This is language-agnostic (the model's own field works on CJK and Latin alike). zh->en: 0 -> 11 streaming commits, first at 6s. en->zh: 7 streaming, first at 6s. 2. split_text_units: CJK char-split fallback. When native stable_text is unavailable, the LCP path now splits CJK runs into character units (one char = one unit ≈ one BPE token) instead of collapsing them. Detection is by Unicode range (not a lang switch), applying to zh/ja/ko equally. 3. Token-LCP variant (tokenize_fn): commit on the model's own token strings instead of text units. Normalizes CJK (1 char ≈ 1 token) and Latin (subword) to the same token budget. Faster first commit (4s) but fragments mid-word; kept as an option, not the default. No language-specific switches. The only lang-aware code is CJK Unicode-range detection in the fallback char-split.
e94d525 to
7610b2a
Compare
All sixteen reports of the M5 screening, measured in one serial session on AC power with the same commits, models and corpus as the partial September series. The September directory is kept unchanged. Native Qwen (#425) fails every comparison: its WER/CER is worse than both Whisper small and Qwen Metal, EOF is slower, and the ten-minute stream fails reproducibly on a 160-sample tail at a pause. Nemotron (#426) passes French against both baselines but fails English against Whisper and Chinese against Metal on quality.
|
Sorry for the long silence on this one. I measured it against the existing backends on an M5 (the full series is in #444) and ran it on real audio, and it is not mergeable yet. The integration itself is clean: the adapter is small, the only conflict with main is Measurements (#444, 90 FLEURS clips per language, three passes)
EOF p95 is also slower than both baselines (0.9 to 1.9 s). Since Metal uses the same model revision, the gap comes from the adapter or the incremental finalizer, not the model. In French it is mostly insertions and deletions: 441 and 330, against 81 and 84 for Metal. What I see on real audio On I traced the repetition to Two other things:
What would make it mergeable
The first one may be better fixed upstream in mlx-qwen3-asr. If you don't have time for this now, no problem, tell me and I'll mark the PR as on hold rather than leave it open without an answer. |
ef7a966 to
bc9e18f
Compare
# Conflicts: # uv.lock
… 0.4.4 0.4.x replaced the finish-time tail refinement pass with a window re-decode that includes the pending tail, so a segment's text is produced once and the adapter commits only what clients do not already hold. Adds an agreed-prefix fallback while stable_text is empty, repairs a non-prefix or rewriting candidate into an append (upstream QuentinFuxa#31) instead of aborting, and ends a segment whose leftover audio is below the log-mel floor instead of raising.
bc9e18f to
d538a8e
Compare
|
|
|
Big improvement, thanks. I checked it on the same manifest (one paced pass, One bug is left before I can merge: revisions near the start of a segment still duplicate text. On The trace on the French recording: The model only capitalized the first word. Since clients cannot take back text, I'd suggest appending only what goes beyond what they already hold: when the candidate rewrites the source and is at least as long, emit With that fixed I'm happy to merge: French beats both baselines, Chinese matches Metal by your numbers, and English is close to Whisper small. |
A window re-decode can capitalise the first word of the same sentence. The shared-prefix
comparison was case-sensitive, so a candidate that only changed 'là' to 'Là' reported no shared
prefix, and the whole segment reached clients a second time ('là il y a eu telLà il y a eu tel
second de silence').
Comparing with casefold makes that candidate extend the committed text, so only the tail is sent.
Slice at the shared prefix rather than at len(source): slicing at len(source) drops the first
letters of a word when a rewrite cuts inside one, which broke the upstream QuentinFuxa#31 cases.
Tests replay the reported trace and the fr_1935 form of it, and assert that what clients hold plus
the delta reconstructs the candidate when a rewrite cuts inside a word.
|
Fixed in The cause was the case-sensitive shared-prefix comparison. Your candidate only capitalises the One deviation from your suggestion: I did not slice at Tests replay your exact trace, the What I could not do: reproduce the 7 of 30 on French clips. |
Both PRs add a backend next to the same registries, so each conflict keeps both sides: pyproject extras and [tool.uv] conflicts now list mlx-qwen3-asr and nemotron-mlx-asr, benchmark/compat.py maps both, parse_args.py offers both in --backend, and backend_support.py exports both availability helpers. uv.lock regenerated with UV_NO_CONFIG=1 (408 packages).
Add an optional
mlx-qwen3-asradapter using the native incremental stream. Model loading and warmup happen once at startup; decoding runs in WLK's counted inference executor under a shared lock, with separate state and language/context options for each session.Session language validation uses the model’s declared 30-language inventory; unsupported Whisper languages are rejected. Stable text is committed directly. Pauses, speaker changes and EOF drain
finish_streaming()without a second full-utterance decode. Preserve spaces and reject revisions of an already confirmed prefix instead of duplicating text. The proposed generic ASR transform layer, timestamp wrapper and their large unit-test suite are removed. Timestamps describe chunk bounds, not word alignments.Validation: The final head passes all required CI checks, including real-audio boundary scenarios; three adapter scenarios cover boundaries, session isolation and failed prefix revisions. A frozen extra installation on M5 completed real English/French/Chinese streams. Ruff and lock checks passed. The lock adds the candidate package without upgrading existing package versions.
The evaluated head (
24082a0) includes explicit pipeline failure reporting and schema-3.1 packet pacing from #443. A real shared-model check also matches sequential and concurrent English/Chinese transcripts exactly.The completed M5 evaluation does not support merging this head. All sixteen reports were measured on AC power (#444,
benchmarks/evaluations/m5-20261107/). English/French WER is 27.65%/44.68% and Chinese CER 28.46%: 16.0/27.0/2.7 points worse than Whisper small and 6.3/20.0/16.6 points worse than Qwen Metal with the same model revision. EOF p95 is slower than both baselines in every pass, and the MLX allocator peak is higher. Most of the gap with Metal is insertions and deletions: hypotheses drop spans at segment boundaries and repeat others, which points to the adapter or incremental finalizer rather than the model.The ten-minute Chinese stream fails reproducibly after 101.5 s: at a pause,
start_silence()callsfinish_streaming()with a 160-sample tail and mlx-qwen3-asr 0.3.5 raisesAudio too short for log-mel extraction. The restored README, badge and published figures are unchanged.