Skip to content

feat: wake parked subscribers on context cancellation - #11

Closed
VojtechVitek wants to merge 4 commits into
masterfrom
subscriber-wake-on-ctx-cancel
Closed

VojtechVitek wants to merge 4 commits into
masterfrom
subscriber-wake-on-ctx-cancel

Conversation

@VojtechVitek

@VojtechVitek VojtechVitek commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two bugs, one old and one mine:

  1. Lost-wakeup race (pre-existing, the CI flake): Write()/Close() called cond.Broadcast() without holding rb.mu. A reader checks writePos/closed under rb.mu, then parks in sync.Cond.Wait() — which adds it to the notify list only after the check. A broadcast landing in that window fires at zero waiters → the reader parks forever. Intermittent (~1/15 locally, TestSubscriberDrainsClosedStream in CI), reproduced on base master.
  2. The eager context.AfterFunc in the first iteration of this PR: dropped stop() leaked parent-ctx child-map entries for subscribers that outlive their registration (EOF, ErrTooSlow), and a mass cancel ran one broadcast per canceled subscriber — O(N²) re-parks serialized on the same rb.mu the fixed Write() needs.

Both found by peer review (two independent reviewers); both fixed here.

Fix 1: hold rb.mu around store/close + Broadcast

Write() and Close() now broadcast under rb.mu (the fill loop stays lock-free). This restores the sync.Cond invariant: no broadcast can land between a reader's re-check and its park.

Measured (full suite, M4 Pro):

Benchmark pre-fix post-fix
Write single item, 1–20k readers 3.22–3.32 ns 4.04–4.51 ns (+27%, 0 allocs)
Write batch (1k–1M) 54.3–55.6 ns 54.2–55.3 ns (unchanged)
Fan-out throughput, 10k subs 100.7M reads/s 100.7M reads/s (unchanged)

Correctness beats nanoseconds: there is no way to keep Write fully lock-free with sync.Cond — the predicate-mutation-under-mu invariant is structural. The mutex is forever uncontended (single writer).

Fix 2: WakeOnCancel is a shared reaper, not per-subscriber

One reaper goroutine per RingBuffer, ticking every StreamReaperInterval (default 250 ms) only while at least one subscriber opted in, broadcasting under rb.mu — so canceled parked readers exit via the existing checkEnd path within one interval. Zero per-subscriber goroutines, zero parent-ctx map entries, no cancel storm. A subscriber releases on its terminal Read() error or via the new idempotent Close(); the reaper stops when the last opted-in subscriber is gone.

Measured (BenchmarkSubscribe):

WakeOnCancel ns/op B/op allocs/op
off (default) ~55 192 3
on ~84 304 4

The extra alloc is cancelCtx.Done()'s lazy channel from the ctx.Done() != nil guard, not the reaper.

Tests

  • TestReaperStopsAfterLastOptedInSubscriber, TestWakeOnCancelNoReaperByDefault, TestCloseStopsReaperBeforeReadEnds (internal, no sleeps — asserts reaper exit deterministically).
  • Opted-in wake tests now run with a 5 ms interval; default-path tests pin the no-wake default.
  • Full suite + -race -count=2 green; go vet clean.

Caveats (honest)

  • An abandoned opted-in subscriber — one that stops calling Read() without a terminal error and never calls Close() — keeps the reaper alive. Documented on Close().
  • Mass cancel drains within one StreamReaperInterval (250 ms default), not instantly — that is the design's bounded staleness.
  • Benchmark note: count=1 runs are indicative; the single-item Write +27% was reproduced across three runs per tree.

🤖 This message was generated by pi using GLM 5.3 on behalf of Vojtech.

VojtechVitek and others added 4 commits October 26, 2026 16:39
A Subscriber parked in Read() slept in sync.Cond.Wait() and only
observed ctx cancellation via the non-blocking check before the wait.
With no Write/Close calls, a canceled subscriber context never woke a
parked reader -- it could sleep forever. The documented workaround was
to keep calling Write() with zero items.

Register context.AfterFunc(ctx, ...) in Subscribe() to broadcast on ctx
cancellation. The parked reader wakes, loops, and returns ctx.Err()
via the existing checkEnd path.

The Write hot path is untouched (lock-free fill + writePos.Store +
cond.Broadcast, unchanged) -- the cost is one AfterFunc registration
per Subscribe call. Write-path benchmarks unchanged (ns/op within
noise, 0 allocs/op).

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
Per maintainer review: the wake costs one context.AfterFunc goroutine
per subscriber, parked until ctx is done -- too much for a throughput
library to impose by default. Move it behind SubscribeOpts.WakeOnCancel,
default off. Guard ctx.Done() == nil (Background/Todo) even when opted
in. Restore the Write() doc note for the default path: an idle writer
should occasionally call Write() with zero items to drop canceled
subscribers.

Default-path cost: Subscribe 3 allocs/op, 176 B/op (~58 ns), Write
untouched. Opted-in: 9 allocs/op, 696 B/op (~685 ns), one parked
goroutine per subscriber. Benchmarks included.

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
Full-suite benchmarks before/after: no regressions (benchstat all ~,
p=1.000; geomean -0.89% = noise). Document the toggle cost in the
README (new section) and the WakeOnCancel doc comment:

- off (default): ~46 ns/op, 176 B/op, 3 allocs/op per Subscribe
- on: ~566 ns/op, 696 B/op, 9 allocs/op, +1 parked goroutine per
  subscriber until ctx done -- Subscribe-time only, Write untouched

Co-Authored-By: GLM 5.3 <noreply@z.ai>
Two changes from peer review (Opus 5.5 + Muse):

1. Write()/Close() now hold rb.mu around the writePos store / close +
   cond.Broadcast. Before, a broadcast landing between a reader check
   and its sync.Cond.Wait() park fired at zero waiters -- the reader
   parked forever (intermittent CI flake, ~1/15). The fill loop stays
   lock-free. Measured: single-item Write 3.23 -> 4.11 ns/op (+27%, 0
   allocs); batch unchanged; fan-out throughput unchanged at 10k subs.

2. WakeOnCancel no longer registers context.AfterFunc per subscriber --
   the dropped stop func leaked parent-ctx child-map entries for
   subscribers that outlived their registration (EOF, ErrTooSlow), and
   a mass cancel ran one broadcast per canceled subscriber: O(N^2)
   re-parks on the same rb.mu that Write() needs. Instead: one shared
   reaper goroutine per ring buffer, ticking every StreamReaperInterval
   (250ms) while at least one subscriber opted in, broadcasting under
   rb.mu. A subscriber releases on its terminal Read() error or via
   Close(); the reaper stops when the last opted-in subscriber is gone.
   Measured: Subscribe ~84ns/4 allocs on vs ~55ns/3 off (was 566ns/9).

Co-Authored-By: DeepSeek V4.1 Flash <noreply@deepseek.com>
@VojtechVitek

Copy link
Copy Markdown
Contributor Author

I don't like this. Closing.

🤖 The real finding — both flagged the eager AfterFunc as flawed

  • Opus P1: we drop AfterFunc's stop(). A subscriber that exits early (EOF, ErrTooSlow) with a long-lived ctx — the natural 100k-sub pattern — pins its
    registration in the parent ctx's child map forever. Unbounded parent-ctx growth.

  • Muse P1: same leak, plus the mass-cancel storm — 100k cancel callbacks, each taking rb.mu and each Broadcast waking all parked readers: O(N²) re-parks
    serialized on the very mutex Write now needs.

I will come up with a different fix myself.

@VojtechVitek
VojtechVitek deleted the subscriber-wake-on-ctx-cancel branch October 27, 2026 20:25
@VojtechVitek

Copy link
Copy Markdown
Contributor Author

Closing as superseded — splitting into two smaller PRs off master:

  1. Concurrent-safe Write()/Close() (multi-multi): carries this PR's cond-contract mutex (Write/Close broadcast under rb.mu) and extends it over the fill loop, plus Write() returning a new ErrClosed sentinel after Close().
  2. Opt-in SetKeepAlive(interval): replaces the per-subscriber WakeOnCancel reaper here — simpler (one shared ticker per buffer, idle-detecting, zero per-subscriber state, nothing on the Write() path).

The WakeOnCancel design in this PR didn't survive review: the eager context.AfterFunc leaks parent-ctx child-map entries (dropped stop()) and mass-cancels storm O(N^2) on rb.mu. Thanks for the review pressure that surfaced both.

🤖 This message was generated by pi using GLM 5.3 on behalf of Vojtech.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant