Skip to content

Record reproducible M5 backend evaluation and session checks - #444

Merged
QuentinFuxa merged 8 commits into
mainfrom
dev/m5-benchmark-results
Nov 13, 2026
Merged

QuentinFuxa merged 8 commits into
mainfrom
dev/m5-benchmark-results

Conversation

@QuentinFuxa

@QuentinFuxa QuentinFuxa commented Sep 7, 2026 •

Copy link
Copy Markdown
Owner

Retain a reproducible M5 evaluation of #425 and #426 against Whisper small MLX/LocalAgreement and Qwen3 Metal: 30 FLEURS clips per language, three paced passes after warmup, plus a ten-minute Chinese stream per backend. Historical figures, the root README and its badge remain unchanged.

m5-20261107/ holds the complete series: all sixteen schema-3.1 reports, measured in one serial session on AC power (253 power samples, all on AC, no run flagged), with the same commits, model revisions and corpus as September. It adds the series runner, fresh-process startup measurements, the power log, environment versions, the summary and a figure. m5-20260907/ keeps the partial September series unchanged, including its AC-to-battery transition and the #446 direct-translation failures.

Neither candidate passes the agreed rule against both existing backends:

Comparison English French Chinese
Native Qwen vs Whisper small fail fail fail
Native Qwen vs Qwen Metal fail fail fail
Nemotron vs Whisper small fail pass pass
Nemotron vs Qwen Metal pass pass fail
  • Native Qwen: 27.65% / 44.68% WER and 28.46% CER, worse than Whisper small and than Qwen Metal with the same weights; EOF p95 is slower than both in every pass. The ten-minute stream fails reproducibly after 101.5 s when finish_streaming() receives a 160-sample tail at a pause.
  • Nemotron: EOF p95 is 71% to 94% lower than both baselines in every pass and quality matches September exactly, but English WER is 7.6 points worse than Whisper and Chinese CER 10.8 points worse than Metal. Its ten-minute stream completes without a repetition loop.
  • Whisper's ten-minute stream repeats the same phrase 31 times again; Metal's completes without a loop.

The measured revisions are those of the candidate PRs and b2baee9, not current main. Ordinary desktop activity was not controlled. The details and limits are in m5-20261107/METHODS.md and REVIEW.md.

All sixteen reports of the M5 screening, measured in one serial session on AC
power with the same commits, models and corpus as the partial September series.
The September directory is kept unchanged.

Native Qwen (#425) fails every comparison: its WER/CER is worse than both
Whisper small and Qwen Metal, EOF is slower, and the ten-minute stream fails
reproducibly on a 160-sample tail at a pause. Nemotron (#426) passes French
against both baselines but fails English against Whisper and Chinese against
Metal on quality.
@QuentinFuxa
QuentinFuxa marked this pull request as ready for review November 8, 2026 01:54
@QuentinFuxa
QuentinFuxa merged commit 1b2986d into main Nov 13, 2026
9 checks passed
@QuentinFuxa
QuentinFuxa deleted the dev/m5-benchmark-results branch November 13, 2026 19:28
clkao added a commit to clkao/WhisperLiveKit that referenced this pull request Nov 13, 2026
clkao added a commit to clkao/WhisperLiveKit that referenced this pull request Nov 13, 2026
clkao added a commit to clkao/WhisperLiveKit that referenced this pull request Nov 14, 2026
…uentinFuxa#444 native 44.68% / metal 24.68% / whisper-small 17.66%; commit-seam finding
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant