Parallel backend launch is serialized by the launcher's global write lock, so one hung backend blocks every other server #14978

Summary

registerAllToolsParallel starts one goroutine per backend. Each goroutine calls launcher.GetOrLaunch, which goes through syncutil.MapGetOrCreate(&l.mu, l.connections, ...). MapGetOrCreate takes l.mu.Lock() and keeps holding it while it runs create(). create() is the entire launch: docker run, the MCP initialize handshake, and up to startupTimeout of waiting in launchStdioConnection.

Every launch therefore holds the launcher's single RWMutex exclusively. The goroutines run one at a time, even though the gateway logs Parallel server launching enabled (default) and sequentialLaunch=false.

Code references (v0.4.30; unchanged on main / v0.4.33):

• internal/syncutil/cache.go MapGetOrCreate: the write lock is held across create()
• internal/launcher/launcher.go getOrLaunchWithConfig: create() calls launchStdioConnection (or mcp.NewHTTPConnection)
• internal/launcher/launcher.go GetOrLaunchForSession: same pattern. l.mu.Lock() with a deferred unlock wraps launchStdioConnection.
• internal/server/tool_registry.go registerAllToolsParallel → ensureToolsRegistered → launcher.GetOrLaunch

Impact

• Startup time is the sum of all backends' times, not the slowest one. One run with six stdio backends took about 95s: 7.6s + 10.5s + 8.4s + 2.3s + 63s + 2s.
• One hung backend blocks every backend queued behind it, for up to the full startupTimeout. In our failing run, a stdio backend (npx -y @growthbook/mcp@2.2.0 on node:lts-alpine) never finished its handshake. The other four backends never logged Launching new server, including github-mcp-server, which normally connects in about 2s. When the gateway was stopped they all failed together with context canceled after 2m25s.
• /health stays unreachable until every backend has finished. NewUnified calls registerAllTools before the listener is created, so the health probe sees HTTP code 0. With gh-aw's fixed ~149s health-check budget, the whole workflow fails before the agent starts.
• Calls to running servers can block too (from reading the code; not observed). The same lock guards GetOrLaunchForSession, so a tool call to an already-running server can stall behind another server's launch. getServerConfig takes RLock and waits on the held write lock.

Evidence

Successful run, sequentialLaunch=false. Each launch starts only after the previous connection is established:

20:18:26.550 Launching new server: serverID=growthbook
20:18:34.119 Connection established: serverID=growthbook
20:18:34.126 Launching new server: serverID=figma
20:18:44.686 Connection established: serverID=figma
20:18:44.692 Launching new server: serverID=atlassian
20:18:53.119 Connection established: serverID=atlassian
20:18:53.129 Launching new server: serverID=github
20:18:55.378 Connection established: serverID=github
20:18:55.385 Launching new server: serverID=firebase
20:19:58.782 Connection established: serverID=firebase
20:19:58.798 Launching new server: serverID=safeoutputs
20:20:00.914 Connection established: serverID=safeoutputs

Existing tests such as TestRegisterAllToolsParallel_MultipleBackends use fast mock HTTP backends and only check which tools end up registered, so they don't catch the serialization.

Suggested fix

Don't hold l.mu during a launch. Deduplicate concurrent launches per key instead, e.g. golang.org/x/sync/singleflight keyed by serverID (plus sessionID for the session pool), or a per-key in-flight entry/sync.Once. Take l.mu only for short reads and writes of the maps and state (connections, serverStartTimes, serverErrors). recordStart and recordError currently assume they are called with the lock held, so they would need to lock for themselves.

A regression test could add two stdio or HTTP backends whose initialize blocks on a channel. It would then assert that both launches start before either completes, or that total registration time is close to the slower backend's time rather than the sum.