You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
chore(op-node): Fix light-sequencer follow-source ↔ sequencer oscillation during recovery #21964
When an op-node is both a sequencer and a follow-source follower (the "light sequencer" config), recovery from a divergence between its unsafe head and the upstream safe chain oscillates, making net progress of ~1 block per 2 poll cycles (~4s/block on a ~2s poll)
instead of converging directly. This roughly doubles convergence time and wastes a forkchoice reset on every other cycle.
Surfaced while de-flaking TestSupernodeLightSequencerInteropInvalidMessageReplacement (#21806). That PR fixed the test; this node behavior is unchanged and needs its own fix.
Mechanism
FollowSource already forceResets onto the upstream on divergence (the #21119/#21155 fix), but the sequencer rebuilds a block on its own fork before the next poll lands — so the follower re-diverges and gets reset again. Net advance is ~1 block per 2 cycles:
convergence is bounded, not stalled, but runs at half the achievable rate.
Proposed direction
Make the follow-source forceReset resist being clobbered by the next sequencer build during recovery — e.g. hold sequencing until the reorg consolidates (upstream safe matches locally), so the sequencer rebuilds from the settled head rather than racing the next poll.
Should roughly halve convergence time. Not a liveness bug — recovery completes — so this is wasted work, not a stall.
Logging an occurrence that upgrades this from a convergence-time issue to a correctness problem: the oscillation can end with the sync-start recovery path relabeling the local fork as safe/finalized.
Occurrence:memory-all-kona-op-rethjob 5360973, develop commit 12b6e15 (docs-only #21917, unrelated). First failure of TestSupernodeLightSequencerInteropInvalidMessageReplacement since #21806 merged (07c6612, ~90 min earlier). The new gate sys.L2BCL.Reached(safety.CrossSafe, settledBlock, 45) (invalid_message_reorg_test.go:186) passed — yet AssertTxInBlock(settledBlock, settledTxHash) at line 188 failed: block 30 on the light sequencer's EL lacks tx 0x7ed2adff4ad39f0abe7c51a06b7cd004bd84c131902447ada2a65d40960def8c.
Mechanism (step log, 19:32:14 UTC): the usual oscillation (engine resets flip-flopping between follow-source refs and locally-built blocks; op-reth ChainReorged ping-pong 30↔32, common ancestor 17) left the engine with unsafe=7d9ba4:30 (local fork) behind safe=779e1f:32 (upstream):
start.go:166 Loaded current L2 heads unsafe=7d9ba4..f4aae1:30 safe=779e1f..a6c520:32 finalized=4abe80..ab1990:0
start.go:182 ERROR Unsafe head is behind known finalized/safe blocks ... Attempting recovery now.
op-reth: Forkchoice updated head=0x7d9ba4... safe=0x7d9ba4... finalized=0x7d9ba4...
engine_controller.go:1256 Reset of Engine is completed local_unsafe=cross_unsafe=local_safe=cross_safe=finalized=7d9ba4..f4aae1:30
That recovery is FindL2Heads in op-node/rollup/sync/start.go:181-185 (at 12b6e15), which calls L2HeadsForELSyncWithOffset(tip=unsafe); that returns {Unsafe: tip, Safe: derived, Finalized: derived} (start.go:143), and with the default zero OffsetELSafe, derived == tip. So safe and finalized were force-set to the local unsafe fork block 30.
Why the #21806 gate is insufficient: after the force-set, the CL reports cross-safe (and finalized) at height 30 on its own fork, so both the CrossSafe gate and the EL safe check pass at the right height on the wrong chain. Height-based safety-label checks cannot distinguish forks once recovery has relabeled the local fork.
Separate smell: this path marks a locally-built, never-derived fork block as finalized in the engine forkchoice. It's meant for EL-sync bootstrap, but the oscillation makes it reachable in normal light-sequencer operation.
Cross-ref: flake tracker #21635 (this is the first post-#21806 occurrence; paged from the kona variant).
Logging another occurrence of the oscillation defeating the height-based gate.
Occurrence: 2026-08-01, memory-all-opn-op-reth-praguejob 5412656 on develop (job renamed from memory-all-opn-op-reth by #21919's L1-fork matrix on 07-31). Sole failure of 382 tests: TestSupernodeLightSequencerInteropInvalidMessageReplacement — third failure past the #21806 band-aid gate. Same signature as above: labels reached height 31, but the light sequencer's EL block 31 was its own locally-built fork, so AssertTxInBlock missed the settled tx.
Evidence (step log, 20:51 UTC): follow-source reset onto upstream e652d5:32, then the sequencer sealed own-fork block 35 (parent 597995:34) on the very next 2s cadence, followed by the inconsistent-forkchoice deriver reset:
engine_controller.go:1612 WARN [08-01|20:51:31.245] Follow Source: Reorg onto upstream chain component=l2cl-sequencer currentUnsafe=597995..f8f8fa:34 currentSafe=f33493..4b6e7a:29 externalSafe=e652d5..7057c2:32 externalLocalSafe=e652d5..7057c2:32 externalFinalized=6d9417..742075:0
sequencer.go:285 INFO [08-01|20:51:33.246] Sequencer sealed block component=l2cl-sequencer payloadID=0x04b7423ca91fc6a1 block=672731..184135:35 parent=597995..f8f8fa:34 txs=1 time=1,785,617,493
engine_client.go:84 WARN [08-01|20:51:33.252] Failed to share forkchoice-updated signal component=l2cl-sequencer state="&{HeadBlockHash:0x67273164eb8e9dc6e53f981bb2025dd3153d2c6f79ced42210ff7d5132184135 SafeBlockHash:0xe652d59aad55055e32764c8dfe491c0cb0581e08f8a35f97bdf3713cac7057c2 FinalizedBlockHash:0x6d9417469ea85b59b7cfab4ab3c91606c4dfe8c728a83bffcf9438524a742075}" err="Invalid forkchoice state"
sync_deriver.go:227 WARN [08-01|20:51:33.252] Deriver system is resetting component=l2cl-sequencer err="need reset to resolve pre-state problem: pre-block-creation forkchoice update was inconsistent with engine, need reset to resolve: Invalid forkchoice state"
Host health: normal 2s sequencing cadence with ms-range block insert times throughout — not the CI CPU-contention signature. The kona sibling job was green on the same (docs-only) commit.
Activity
claude commentedon Jul 22, 2026
Logging an occurrence that upgrades this from a convergence-time issue to a correctness problem: the oscillation can end with the sync-start recovery path relabeling the local fork as safe/finalized.
Occurrence:
memory-all-kona-op-rethjob 5360973, develop commit 12b6e15 (docs-only #21917, unrelated). First failure ofTestSupernodeLightSequencerInteropInvalidMessageReplacementsince #21806 merged (07c6612, ~90 min earlier). The new gatesys.L2BCL.Reached(safety.CrossSafe, settledBlock, 45)(invalid_message_reorg_test.go:186) passed — yetAssertTxInBlock(settledBlock, settledTxHash)at line 188 failed: block 30 on the light sequencer's EL lacks tx0x7ed2adff4ad39f0abe7c51a06b7cd004bd84c131902447ada2a65d40960def8c.Mechanism (step log, 19:32:14 UTC): the usual oscillation (engine resets flip-flopping between follow-source refs and locally-built blocks; op-reth
ChainReorgedping-pong 30↔32, common ancestor 17) left the engine with unsafe=7d9ba4:30(local fork) behind safe=779e1f:32(upstream):That recovery is
FindL2Headsinop-node/rollup/sync/start.go:181-185(at 12b6e15), which callsL2HeadsForELSyncWithOffset(tip=unsafe); that returns{Unsafe: tip, Safe: derived, Finalized: derived}(start.go:143), and with the default zeroOffsetELSafe,derived == tip. So safe and finalized were force-set to the local unsafe fork block 30.Why the #21806 gate is insufficient: after the force-set, the CL reports cross-safe (and finalized) at height 30 on its own fork, so both the CrossSafe gate and the EL safe check pass at the right height on the wrong chain. Height-based safety-label checks cannot distinguish forks once recovery has relabeled the local fork.
Separate smell: this path marks a locally-built, never-derived fork block as
finalizedin the engine forkchoice. It's meant for EL-sync bootstrap, but the oscillation makes it reachable in normal light-sequencer operation.Cross-ref: flake tracker #21635 (this is the first post-#21806 occurrence; paged from the kona variant).
claude commentedon Aug 1, 2026
Logging another occurrence of the oscillation defeating the height-based gate.
Occurrence: 2026-08-01,
memory-all-opn-op-reth-praguejob 5412656 ondevelop(job renamed frommemory-all-opn-op-rethby #21919's L1-fork matrix on 07-31). Sole failure of 382 tests:TestSupernodeLightSequencerInteropInvalidMessageReplacement— third failure past the #21806 band-aid gate. Same signature as above: labels reached height 31, but the light sequencer's EL block 31 was its own locally-built fork, soAssertTxInBlockmissed the settled tx.Evidence (step log, 20:51 UTC): follow-source reset onto upstream
e652d5:32, then the sequencer sealed own-fork block 35 (parent597995:34) on the very next 2s cadence, followed by the inconsistent-forkchoice deriver reset:Host health: normal 2s sequencing cadence with ms-range block insert times throughout — not the CI CPU-contention signature. The kona sibling job was green on the same (docs-only) commit.
Generated by Claude Code