Repository navigation
feat(eot): feed the local mini model its 2.0 s window and turn length - #2850
Draft
chenghao-mou wants to merge 7 commits into
Draft
chenghao-mou wants to merge 7 commits into
chenghao-mou wants to merge 7 commits into
Conversation
@livekit/local-inference 0.2.8 ships v1.1-mini, which reads the last 2.0 s of audio and takes the turn length, counted from the user's first speech, as a feature. The local transport now buffers 2.0 s and sends turnSamples to the runner; AudioRecognition marks the turn start on VAD start of speech. The per-language thresholds match the Python SDK's v1.1 refit.
chenghao-mou
added
the
review_effort:high
🦋 Changeset detectedLatest commit: 1953fd2 The changes in this PR will be included in the next version bump. This PR includes changesets to release 41 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Rename markSpeechStart(lookback) to onStartOfTurn(startedAt), after the endpointing peer onStartOfSpeech(startedAt). AudioRecognition now calls it once per turn, where it sets userTurnStart, so the transport no longer filters repeated starts and a turn that starts from STT gets a start too. The stream turns the start time into a duration when called, so the channel delay can't stretch the turn.
Both onStartOfTurn methods now take startedAt; the local transport turns it into a sample position when it reads the sentinel. The drain lag this adds is sub-millisecond on an idle loop, and a long stall already delays inference.
The local transport read the clock when it drained the start-of-turn sentinel. During a cloud reconnect the audio channel backs up, so after a fallback to local the turn length was too long by the backlog. The stream now converts the start time to samples when it queues the sentinel, and the transport takes that sample count. - Keep one turn counter in the local transport, and send at least the audio the model reads. - Size the window from the binding's EOT_MAX_SAMPLES, and require turnSamples on the inference request. - Tell the turn detector only a known speech start. A turn opened at end of speech keeps the count from the last flush. - Keep the measured memory figure in one place.
The sentinel carries the elapsed time measured when it is queued, and the local transport converts it to samples. The name no longer collides with the turnSamples sent to predict, which counts up to inference.
6 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The next local mini model (v1.1-mini,
@livekit/local-inference0.2.8) reads the last 2.0 s of audio and takes the turn length as a feature. This ports livekit/agents#8113: the local transport keeps the window the binding reports asEOT_MAX_SAMPLESand sendsturnSamples, counted from the user turn start, whichAudioRecognitionreports once per turn. The per-language thresholds match the Python SDK.Warning
Blocked on the
@livekit/local-inference0.2.8 release.pnpm-lock.yamlstill resolves 0.2.7, which rejectspredict(pcm, turnSamples)and has noEOT_MAX_SAMPLESexport, so CI fails until it ships.Changes Made
LocalTransportbuffersEOT_MAX_SAMPLESand passesturnSamplesto the runner, which forwards it topredict.turnSamplesnever drops below the audio sent.BaseStreamingTurnDetectorStream.onStartOfTurn(startedAt)measures the elapsed time when it queues the sentinel, so a backlog drained after a cloud reconnect does not stretch the turn.AudioRecognitionreports only a known speech start; a turn opened at end of speech counts from the last flush.LOCAL_LANGUAGES; one memory note, ~540 MB measured in Node.Pre-Review Checklist
Testing
restaurant_agent.tsandrealtime_agent.tswork properly (for major changes)Additional Notes
Local checks ran against a 0.2.8 binding built from source.
Addresses AGT-3736
Initial prompt and agent context
Model: Claude Opus 5.5
Then, after the Python change and a threshold refit: "yes, open the PR and open the node js PR too."