Repository navigation
[bugfix] bound connect and read timeouts for Memcached and WebSocket collection - #4493
SiriusTL666 wants to merge 1 commit into
Conversation
…ket collection Both collectors created a bare Socket and called connect(SocketAddress) with no timeout, then read the response with no SO_TIMEOUT. A dropped SYN, a firewalled port, or a peer that accepts the connection but never answers blocks the collector until the OS-level TCP timeout — minutes — while the worker stays occupied and the monitor shows NO DATA. Both classes already had a SocketTimeoutException branch that could never fire, because nothing ever set a timeout. - Pass the monitor's timeout (default 10s) to connect(addr, timeout) and setSoTimeout for both collectors. - Add an optional timeout parameter to app-memcached.yml and app-websocket.yml. - Extend both unit tests to assert the timeout value reaches the socket, once configured and once falling back to the default.
|
Hi maintainers, the workflows on this PR are showing For context: linked issue is #3197, and I checked before starting that no IPMI fix PR existed. I left a note on the issue that I would step back if either of the earlier commenters is already working on it — happy to coordinate. |
|
While waiting for a maintainer to approve the workflows on this PR (this is my first contribution here, so they show https://github.com/SiriusTL666/hertzbeat/actions/runs/40178308108
That covers the Maven build, the frontend release-asset build, and the server release-layout verification, so the change compiles and the module tests pass on the project's own CI definition. To be explicit: this is a run in my fork, not on this PR — it is evidence for the reviewer, not a substitute for the required approval. Once someone approves, the run on this PR should behave the same. Also worth noting: in the fork run I had to touch a real (non-markdown) file to trigger the push-based workflow, since Happy to address any review comments. |
What's changed?
MemcachedCollectImplandWebsocketCollectImplboth create a bareSocketand callconnect(SocketAddress)with no timeout, then read the peer's reply with noSO_TIMEOUT. When the SYN is dropped, the port is firewalled, or the peer accepts the connection but never answers, the call blocks until the OS-level TCP timeout — typically minutes — while the collection worker stays occupied and the monitor reports NO DATA.Both classes already had a
catch (SocketTimeoutException)branch, but nothing ever set a timeout, so that branch was unreachable.Changes
socket.connect(address, timeout)plussocket.setSoTimeout(timeout).MemcachedProtocolandWebsocketProtocolgain an optionaltimeout, exposed as a monitor parameter inapp-memcached.ymlandapp-websocket.yml(range 1000–120000 ms, default 10000).connect(...)andsetSoTimeout(...), and one asserting the 10s fallback. The WebSocket cases also assert the unreachable branch now reportsUN_CONNECTABLE.This is the same class of defect as #4492 (SSL certificate collection) and #4490 (IPMI UDP receive). I audited the rest of
hertzbeat-collector-basicforSocket/DatagramSocket/DatagramChannelusage:UdpCollectImplalready setsSO_TIMEOUT, and the SSH/FTP collectors use libraries that take a timeout argument, so these two were the remaining gaps.Checklist
Verification
mvn -pl hertzbeat-collector/hertzbeat-collector-basic test -Dtest=MemcachedCollectImplTest,WebsocketCollectImplTest— 8/8 pass (5 memcached, 3 websocket).checkstyle:checkonhertzbeat-collector-basicandhertzbeat-common-core— 0 violations.No issue exists for this one; like #4492 I found it while auditing for unbounded network waits. Happy to file issues first if the project prefers that.
Add or update API
No HTTP API change. The
timeoutmonitor parameter is optional and defaults to 10s, which is bounded where the previous behaviour was unbounded.