hiero-ledger / hiero-ledger/hiero-consensus-node

Network errors : not reported properly, cause huge CPU usage

Open
#18,682 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
406
Forks
226
Avg merge
3d 4h
Merged PRs (30d)
210

Description

Found the situation on multi-days NFT test run, thanks to @OlegMazurov .

In case if HAproxy pod died (OOMKilled due to another issues),
2 problems emerge that complicate debugging/ruin functionality:

1)
CN node does not report such network error properly to make root-case analysis efficient (no traces in the log!)

Situation, there was network glitch :
kubectl -n solo-alex-kuzmin-n0 get pods
NAME READY STATUS RESTARTS AGE
envoy-proxy-node1-69c5bcdc4c-p8lqp 1/1 Running 0 26h
...
haproxy-node9-6666c79b4f-25bqn 0/1 ContainerStatusUnknown 1 (26h ago) 26h
haproxy-node9-6666c79b4f-ldm27 1/1 Running 1 (14h ago) 14h <-- root cause
minio-pool-1-0 0/2 Pending 0 14h
network-node1-0 6/6 Running 1 (13h ago) 26h
...
network-node9-0 6/6 Running 1 (13h ago) 26h
nlg-network-load-generator-5db847f8cc-ss7hb 1/1 Running 0 26h

corresponding CN node was hosed withut any clues in logs re network error.

Further investigation revealed that
this code could be a culprit:
https://github.com/hiero-ledger/hiero-consensus-node/blob/4440cd414412b5f2b7dd5d53a398bd4e519fe9b3/platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/network/connectivity/OutboundConnectionCreator.java#L123
} catch (final SocketTimeoutException | SocketException e) {
NetworkUtils.close(clientSocket, dis, dos);
logger.debug(
TCP_CONNECT_EXCEPTIONS.getMarker(), "{} failed to connect to {} with error:", selfId, otherId, e);
// ConnectException (which is a subclass of SocketException) happens when calling someone
// who isn't running yet. So don't worry about it.
// Also ignore the other socket-related errors (SocketException) in case it times out while
// connecting.
} catch (final IOException e) {
NetworkUtils.close(clientSocket, dis, dos);
// log the SSL connection exception which is caused by socket exceptions as warning.

2) CN code spikes CPU usage practically killing other threads

Contributor guide

Open the contributing guide

Research direction

Start with platform-sdk/swirlds-platform-core/src/main/java/com/swirlds/platform/network/connectivity/OutboundConnectionCreator.java around line 123, including both exception handlers. Trace how failed HAProxy connections are handled and logged, then investigate the reported CPU spike during the same failure scenario. Done means network failures provide useful log evidence without causing the CN node's other threads to be overwhelmed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.