[clkao]

feat: MLX-native sortformer diarization backend (pure MLX, no NeMo)

Summary

The existing sortformer diarization backend requires NeMo plus a .nemo
model download, which is heavy and fails offline. This PR adds
mlx-sortformer: the same streaming Sortformer model running through
mlx-audio's MLX conversion
(mlx-community/diar_streaming_sortformer_4spk-v2.1-fp16), entirely on
Apple Silicon, with no NeMo, no ONNX, and no network access after the
first cache.

The backend mirrors the NeMo backend's contract exactly
(insert_audio_chunk / insert_silence / diarize / get_segments /
close), so the pipeline's diarization loop treats it as a peer. It is
selected with --diarization-backend mlx-sortformer and wired in the same
two places as the other backends (engine init and the per-session
factory). Speaker capping (sortformer_max_speakers) works the same as
for the NeMo backend.

mlx-audio is an optional extra (diarization-mlx-sortformer, Apple
Silicon only) and is imported inside the model loader, so the package
imports cleanly without it.

User impact

• New backend choice; no existing behavior changes. sortformer and
  diart behave exactly as before, including the sortformer_max_speakers
  validation (now also accepting mlx-sortformer, since the MLX backend
  applies the same cap).
• On Apple Silicon with the extra installed, diarization runs fully
  offline on MLX.

Validation

• New tests (tests/test_diarization_mlx_backend.py, hermetic: the
  mlx-audio import boundary is faked, no model download): the streaming
  contract (chunk accumulation, feed at chunk size, segment accumulation,
  max-speaker filter, close), the lazy-import property, config validation
  parity, and the factory wiring. 8 tests.
• Live smoke on Apple Silicon with the real model: the full 31.5s zh→en
  test clip processed at RTF 0.024 through the ported contract, 12
  segments, single speaker correctly identified end to end.
• uv run pytest -q tests/ --ignore=tests/test_pipeline.py: failure and
  error set identical to main (compared run-to-run in the same checkout;
  the pre-existing canary, deepgram, and qwen3-shim failures are
  environmental). The new tests all pass.
• ruff check .: passes.

Checklist

• I searched for an existing issue or discussion and linked it when
  relevant. (No existing issue covers an MLX-native diarization
  backend.)
• I added or updated tests for behavior changes.
• I updated documentation for user-facing changes. (The backend is
  discoverable through --diarization-backend --help; a README mention
  can follow.)
• I ran ruff check ..
• I ran the relevant pytest suite.
• I did not commit credentials, model weights, generated caches, or
  private data.

[QuentinFuxa]

Nice to have a NeMo-free option on Mac, thanks. I ran it against current main (with #461 merged) and there are a few problems in how it plugs into the pipeline:

• diarization_processor checks hasattr(self.diarization, "buffer_audio") to pick the buffered path (Sortformer) vs the cumulative one (Diart). This class uses _buffer_audio, so it takes the Diart path: state.new_diarization gets replaced instead of extended, segments not consumed yet are lost whenever diarize() returns [], and there is no drain at EOF, so the last few seconds are never diarized.
• insert_silence() is a no-op. Only voiced audio is sent to diarization, so speaker timestamps drift away from the transcript after each pause. The NeMo backend shifts global_time_offset for that.
• model.feed() runs on the event loop. fix: run Sortformer inference off the event loop #461 just moved the NeMo forward pass off the loop for this reason. MLX has to stay on one thread (we hit There is no Stream(gpu, ...) in current thread in Add a Nemotron MLX adapter using upstream streaming caches #426), so it probably needs a dedicated thread rather than a plain to_thread.
• The extra pins mlx-audio>=0.4.4,<0.5, and Add a Nemotron MLX adapter using upstream streaming caches #426 needs >=0.5.1. Could you check if the Sortformer conversion works on 0.5.x?

A test going through AudioProcessor with a fake model would have caught the first two.

[QuentinFuxa]

Friendly ping on the review above. The mlx-audio range is the point that interacts with #426: if the Sortformer conversion works on mlx-audio 0.5.x, both backends can share one range and be installed together. The buffered-path detection (buffer_audio), insert_silence() and running feed() off the event loop are still needed before merging.

[clkao]

@QuentinFuxa I probably don't have bandwidth to seriously review what the agents change addressing the feedback and rebase. happy to update the PR but i also don't want to irresponsibly ship it.

[QuentinFuxa]

Thanks for being upfront about that, I prefer it to a rushed update. Let's put this on hold: no pressure, and I won't ping again. I'm converting it to draft so the status is clear. Mark it ready again whenever you or someone else picks it up.

For that day: the four points above still apply, and for the dependency range, #426 will most likely land with mlx-audio>=0.5.1,<0.6, so that is the range to target.