Skip to content

chore(op-node): Fix light-sequencer follow-source ↔ sequencer oscillation during recovery #21964

Open

Description

@op-will

Summary

When an op-node is both a sequencer and a follow-source follower (the "light sequencer" config), recovery from a divergence between its unsafe head and the upstream safe chain oscillates, making net progress of ~1 block per 2 poll cycles (~4s/block on a ~2s poll)
instead of converging directly. This roughly doubles convergence time and wastes a forkchoice reset on every other cycle.

Surfaced while de-flaking TestSupernodeLightSequencerInteropInvalidMessageReplacement (#21806). That PR fixed the test; this node behavior is unchanged and needs its own fix.

Mechanism

FollowSource already forceResets onto the upstream on divergence (the #21119/#21155 fix), but the sequencer rebuilds a block on its own fork before the next poll lands — so the follower re-diverges and gets reset again. Net advance is ~1 block per 2 cycles:
convergence is bounded, not stalled, but runs at half the achievable rate.

Proposed direction

Make the follow-source forceReset resist being clobbered by the next sequencer build during recovery — e.g. hold sequencing until the reorg consolidates (upstream safe matches locally), so the sequencer rebuilds from the settled head rather than racing the next poll.
Should roughly halve convergence time. Not a liveness bug — recovery completes — so this is wasted work, not a stall.

References

Activity

  1. claude commented on Jul 22, 2026

    @claude
    Contributor

    Logging an occurrence that upgrades this from a convergence-time issue to a correctness problem: the oscillation can end with the sync-start recovery path relabeling the local fork as safe/finalized.

    Occurrence: memory-all-kona-op-reth job 5360973, develop commit 12b6e15 (docs-only #21917, unrelated). First failure of TestSupernodeLightSequencerInteropInvalidMessageReplacement since #21806 merged (07c6612, ~90 min earlier). The new gate sys.L2BCL.Reached(safety.CrossSafe, settledBlock, 45) (invalid_message_reorg_test.go:186) passed — yet AssertTxInBlock(settledBlock, settledTxHash) at line 188 failed: block 30 on the light sequencer's EL lacks tx 0x7ed2adff4ad39f0abe7c51a06b7cd004bd84c131902447ada2a65d40960def8c.

    Mechanism (step log, 19:32:14 UTC): the usual oscillation (engine resets flip-flopping between follow-source refs and locally-built blocks; op-reth ChainReorged ping-pong 30↔32, common ancestor 17) left the engine with unsafe=7d9ba4:30 (local fork) behind safe=779e1f:32 (upstream):

    start.go:166  Loaded current L2 heads  unsafe=7d9ba4..f4aae1:30 safe=779e1f..a6c520:32 finalized=4abe80..ab1990:0
    start.go:182  ERROR Unsafe head is behind known finalized/safe blocks ... Attempting recovery now.
    op-reth: Forkchoice updated  head=0x7d9ba4... safe=0x7d9ba4... finalized=0x7d9ba4...
    engine_controller.go:1256  Reset of Engine is completed  local_unsafe=cross_unsafe=local_safe=cross_safe=finalized=7d9ba4..f4aae1:30
    

    That recovery is FindL2Heads in op-node/rollup/sync/start.go:181-185 (at 12b6e15), which calls L2HeadsForELSyncWithOffset(tip=unsafe); that returns {Unsafe: tip, Safe: derived, Finalized: derived} (start.go:143), and with the default zero OffsetELSafe, derived == tip. So safe and finalized were force-set to the local unsafe fork block 30.

    Why the #21806 gate is insufficient: after the force-set, the CL reports cross-safe (and finalized) at height 30 on its own fork, so both the CrossSafe gate and the EL safe check pass at the right height on the wrong chain. Height-based safety-label checks cannot distinguish forks once recovery has relabeled the local fork.

    Separate smell: this path marks a locally-built, never-derived fork block as finalized in the engine forkchoice. It's meant for EL-sync bootstrap, but the oscillation makes it reachable in normal light-sequencer operation.

    Cross-ref: flake tracker #21635 (this is the first post-#21806 occurrence; paged from the kona variant).

  2. claude commented on Aug 1, 2026

    @claude
    Contributor

    Logging another occurrence of the oscillation defeating the height-based gate.

    Occurrence: 2026-08-01, memory-all-opn-op-reth-prague job 5412656 on develop (job renamed from memory-all-opn-op-reth by #21919's L1-fork matrix on 07-31). Sole failure of 382 tests: TestSupernodeLightSequencerInteropInvalidMessageReplacement — third failure past the #21806 band-aid gate. Same signature as above: labels reached height 31, but the light sequencer's EL block 31 was its own locally-built fork, so AssertTxInBlock missed the settled tx.

    Evidence (step log, 20:51 UTC): follow-source reset onto upstream e652d5:32, then the sequencer sealed own-fork block 35 (parent 597995:34) on the very next 2s cadence, followed by the inconsistent-forkchoice deriver reset:

    engine_controller.go:1612  WARN [08-01|20:51:31.245] Follow Source: Reorg onto upstream chain component=l2cl-sequencer currentUnsafe=597995..f8f8fa:34 currentSafe=f33493..4b6e7a:29 externalSafe=e652d5..7057c2:32 externalLocalSafe=e652d5..7057c2:32 externalFinalized=6d9417..742075:0
    sequencer.go:285           INFO [08-01|20:51:33.246] Sequencer sealed block                   component=l2cl-sequencer payloadID=0x04b7423ca91fc6a1 block=672731..184135:35 parent=597995..f8f8fa:34 txs=1 time=1,785,617,493
    engine_client.go:84        WARN [08-01|20:51:33.252] Failed to share forkchoice-updated signal component=l2cl-sequencer state="&{HeadBlockHash:0x67273164eb8e9dc6e53f981bb2025dd3153d2c6f79ced42210ff7d5132184135 SafeBlockHash:0xe652d59aad55055e32764c8dfe491c0cb0581e08f8a35f97bdf3713cac7057c2 FinalizedBlockHash:0x6d9417469ea85b59b7cfab4ab3c91606c4dfe8c728a83bffcf9438524a742075}" err="Invalid forkchoice state"
    sync_deriver.go:227        WARN [08-01|20:51:33.252] Deriver system is resetting              component=l2cl-sequencer err="need reset to resolve pre-state problem: pre-block-creation forkchoice update was inconsistent with engine, need reset to resolve: Invalid forkchoice state"
    

    Host health: normal 2s sequencing cadence with ms-range block insert times throughout — not the CI CPU-contention signature. The kona sibling job was green on the same (docs-only) commit.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-op-nodeArea: op-nodeM-needs-triageMeta: this issue needs to be labelled

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions