hiero-ledger / hiero-ledger/hiero-consensus-node
Network errors : not reported properly, cause huge CPU usage
- Dominant language
- Java
- Stars
- 406
- Forks
- 226
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 210
Description
Found the situation on multi-days NFT test run, thanks to @OlegMazurov .
In case if HAproxy pod died (OOMKilled due to another issues),
2 problems emerge that complicate debugging/ruin functionality:
1)
CN node does not report such network error properly to make root-case analysis efficient (no traces in the log!)
Situation, there was network glitch :
kubectl -n solo-alex-kuzmin-n0 get pods
NAME READY STATUS RESTARTS AGE
envoy-proxy-node1-69c5bcdc4c-p8lqp 1/1 Running 0 26h
...
haproxy-node9-6666c79b4f-25bqn 0/1 ContainerStatusUnknown 1 (26h ago) 26h
haproxy-node9-6666c79b4f-ldm27 1/1 Running 1 (14h ago) 14h <-- root cause
minio-pool-1-0 0/2 Pending 0 14h
network-node1-0 6/6 Running 1 (13h ago) 26h
...
network-node9-0 6/6 Running 1 (13h ago) 26h
nlg-network-load-generator-5db847f8cc-ss7hb 1/1 Running 0 26h
corresponding CN node was hosed withut any clues in logs re network error.
Further investigation revealed that
this code could be a culprit:
https://github.com/hiero-ledger/hiero-consensus-node/blob/4440cd414412b5f2b7dd5d53a398bd4e519fe9b3/platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/network/connectivity/OutboundConnectionCreator.java#L123
} catch (final SocketTimeoutException | SocketException e) {
NetworkUtils.close(clientSocket, dis, dos);
logger.debug(
TCP_CONNECT_EXCEPTIONS.getMarker(), "{} failed to connect to {} with error:", selfId, otherId, e);
// ConnectException (which is a subclass of SocketException) happens when calling someone
// who isn't running yet. So don't worry about it.
// Also ignore the other socket-related errors (SocketException) in case it times out while
// connecting.
} catch (final IOException e) {
NetworkUtils.close(clientSocket, dis, dos);
// log the SSL connection exception which is caused by socket exceptions as warning.
2) CN code spikes CPU usage practically killing other threads
Contributor guide
Research direction
Start with platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/network/connectivity/OutboundConnectionCreator.java around line 123, including both exception handlers. Trace how failed HAProxy connections are handled and logged, then investigate the reported CPU spike during the same failure scenario. Done means network failures provide useful log evidence without causing the CN node's other threads to be overwhelmed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100