[dingzhiqiang]

Description

Add opt-in backend load balancing to the v1 SGLang rollout path. New sessions select the healthy backend with the lowest outstanding execution count normalized by capacity; subsequent requests keep the same backend for cache locality. DataProxy ownership is unchanged, and generation traffic goes directly to the selected backend.

Enable with rollout.backend_routing.enabled: true in v1 single-controller mode. The controller owns the shared router lifecycle, including train/eval sender fencing, draining, and weight-version publication. Metadata retries are idempotent; ambiguous generation failures fail the pool instead of replaying an uncertain execution. Pauses do not consume the backend-health timeout budget, and stale health responses cannot override a resumed pool.

Related Issue

No linked issue.

Type of Change

• ⚡ Performance improvement

Validation

• CPU tests: 226 passed across backend routing, controller lifecycle, recovery, existing v2 router behavior, and OpenAI proxy compatibility.
• python -m pytest -q tests/v2/inference_service/test_backend_router.py tests/test_backend_routing_lifecycle.py tests/test_recover.py: 105 passed.
• python -m pytest -q tests/v2/inference_service/test_router.py tests/experimental/openai/test_proxy_rollout_server.py: 112 passed.
• Workflow executor filtering and dump compatibility: 9 passed.
• Full pre-commit run --all-files passed, including generated CLI documentation; normal commit hooks and Conventional Commit validation also passed.
• Full documentation build was attempted with ./docs/build_all.sh; the existing offline environment lacks the pinned Megatron-Bridge wheel. Markdown formatting and CLI-document generation are checked by pre-commit.
• Multi-node integration was exercised in the experimental source branch; the cleaned and rebased PR commit has CPU regression coverage, but has not been rerun on a GPU cluster.

Performance observation

An asynchronous SWE multimodal comparison used 64 task IDs, 16 TP=4 inference replicas, fixed model weights (lr=0), and nine updates per arm:

Metric	Routing off	Routing on
Sum of rollout durations	21,727.7 s	20,400.7 s
Generated output tokens	62.2066 M	62.2109 M
Per-backend output-token coefficient of variation	0.1918	0.0657

Observed asynchronous rollout duration decreased by 6.1%. This is one sequential OFF/ON comparison, not a guaranteed speedup: asynchronous sample consumption, trajectories, uncached prompt work, and environment latency differ. The first eight updates improved by 2.3%; the ninth contributes most of the aggregate saving. Training-time reductions are excluded from this claim.

Synchronous rollout: no reliable numerical estimate is claimed. A synchronous batch waits for its slowest trajectory, so the asynchronous 6.1% is not a synchronous estimate or an upper bound. The earlier synchronous run was dominated by environment timeouts and cannot establish a comparable scheduling benefit.

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; focused tests added for new functionality
• Documentation updated and built with ./docs/build_all.sh (CLI docs updated; full build blocked as described above)
• Branch is up to date with main
• Self-reviewed via the /review-pr workflow with an infra expert; no blocking findings
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

Additional Context

Routing is disabled by default. Supported scope is v1, SGLang, and single-controller operation. Router state is in-memory for the run; an uncertain execution or router failure requires restarting the experiment. Experimental scheduling policies, training-engine fixes, private launch recipes, and diagnostic instrumentation are excluded.

[sitabulaixizawaluduo]

There seems to be no deletion mechanism, which poses a risk of memory leaks.

[sitabulaixizawaluduo]

If the tail of self.address has a "/", it will access the wrong address.