Improve pre-handshake connection handling

Summary

After a TRON node accepts an inbound TCP connection, the connection is not counted toward the node's existing connection limits (maxConnections and maxConnectionsWithSameIp) until a complete protocol frame has been decoded and the connection reaches ChannelManager.processPeer().

The existing global and per-IP connection limits are not enforced at the accept stage, and there is no dedicated handshake timeout for connections that have not completed the handshake. As a result, incomplete connections may remain open for an extended period, consuming file descriptors and pre-handshake buffer memory.

Introducing admission control for pre-handshake connections and a dedicated handshake timeout would allow these incomplete connections to be reclaimed more promptly, improving connection-management stability and resource utilization.

Root Cause

Existing connection limits and related checks are enforced only after protocol handshake completion, specifically inside ChannelManager.processPeer(). This method is reached only after a complete frame has been decoded and passed through the handshake handling path.
Consequently:

• Every accepted TCP connection proceeds to channel initialization without being checked against the existing global or per-IP connection limits at the accept stage. Connections that have not completed the handshake are therefore not counted toward these limits.
• There is no dedicated handshake timeout. The only relevant timer on the connection is a 60-second ReadTimeoutHandler. Because inbound read activity prevents the read timeout from expiring, a connection can remain open without completing the handshake by occasionally sending small amounts of data.
• The frame decoder can buffer incomplete pre-handshake data up to the transport frame limit (MAX_MESSAGE_LENGTH), allowing an incomplete connection to retain a relatively large amount of buffer memory.

Because existing limits (maxConnections and maxConnectionsWithSameIp) only govern fully established connections inside processPeer(), they leave pre-handshake connections completely unconstrained.

Reproduction

• Establish multiple TCP connections to the node's P2P listening port, which is 18888 by default.
• Keep the connections in the pre-handshake state by not sending a complete handshake frame. Alternatively, periodically send small amounts of data that are insufficient to complete a valid frame.
• Observe that these incomplete connections are not counted toward maxConnections or maxConnectionsWithSameIp, because those checks are reached only after a complete frame has been decoded and the connection enters the handshake handling path.
• Also observe that the connections are not reclaimed by ReadTimeoutHandler as long as intermittent inbound read activity continues.
• As the number of incomplete connections increases, observe the corresponding increase in open file descriptors and pre-handshake buffer memory usage.

Impact

• Incomplete handshake connections are not constrained by the existing global or per-IP connection limits and may accumulate over time.
• Increased file-descriptor and memory usage may reduce the P2P listener's ability to accept legitimate peers, decreasing connection reliability under high connection pressure.
• The impact is limited to connection management, resource consumption, and node availability under connection pressure. The node can recover after the affected connections are released, and the issue does not affect persistent data, consensus correctness, or asset security.
• The number of such connections is ultimately constrained by operating-system and process resource limits, including file-descriptor and available-memory limits.

Suggested Fix

Add lightweight admission control and timeout management for inbound connections before the handshake is completed:

• Pre-handshake admission control: At accept time (channelActive), enforce dedicated limits for pending connections (e.g., maxPendingConnections and maxPendingConnectionsWithSameIp), completely independent of the established connection limits. Register and evaluate the connection before reading any application data so that incomplete connections are included in connection accounting before entering the normal peer-management path.
• Dedicated handshake timeout: Add a fixed handshake deadline for inbound connections. Close any connection that does not complete the handshake within the configured time window, regardless of intermittent inbound read activity. This timeout should be independent of ReadTimeoutHandler and should not be extended by arbitrary incoming bytes.
• Pre-handshake frame-size limit: Apply an appropriate frame-size limit before handshake completion based on the maximum valid handshake message size, rather than allowing an incomplete connection to buffer data up to the full transport-frame limit (MAX_MESSAGE_LENGTH). After the handshake succeeds, the normal transport-frame limit can be applied.
• Pending-slot cleanup: Release the corresponding pending-connection slot whenever the channel closes, the handshake fails, or the handshake completes and the connection transitions to the normal peer-management path. Ensure that cleanup is performed exactly once so that pending-connection counters cannot leak or be decremented multiple times.

Tests

The fix should include regression tests covering the following scenarios:

• Global pending-connection limit: Open pre-handshake connections until the global pending limit is reached and verify that additional inbound connections are rejected or closed without entering the normal peer-management path.
• Per-IP pending-connection limit: Open multiple pre-handshake connections from the same source IP and verify that the per-IP pending limit is enforced independently of the normal maxConnectionsWithSameIp check.
• Handshake timeout: Keep a connection open without completing the handshake and periodically send small amounts of data. Verify that the connection is still closed when the dedicated handshake deadline expires.
• Successful handshake: Verify that a connection completing the handshake within the deadline is removed from the pending pool and transitions correctly to the existing connection-management logic.
• Pending-slot cleanup: Verify that pending counters are released correctly when a channel closes before handshake completion, when handshake validation fails, and when the handshake succeeds.
• Pre-handshake frame-size limit: Send a pre-handshake frame whose declared or accumulated size exceeds the dedicated pre-handshake limit, and verify that the connection is closed before it can buffer data up to MAX_MESSAGE_LENGTH.

General Principle

Connection accounting should cover the entire lifetime of an inbound connection, including the period between TCP accept and protocol handshake completion.

Read activity alone should not be treated as evidence that an incomplete connection is making valid handshake progress. Pre-handshake connections should therefore have independent admission limits, a fixed handshake deadline, and resource bounds appropriate to the small amount of data required to complete the handshake.

The goal is to prevent incomplete connections from consuming disproportionate resources while keeping the pre-handshake path lightweight and avoiding unnecessary complexity in the normal peer-management flow.

Start at inbound channelActive handling and ChannelManager.processPeer(), then trace the existing maxConnections, maxConnectionsWithSameIp, ReadTimeoutHandler, and MAX_MESSAGE_LENGTH logic. Add regression coverage for pending limits, timeout, handshake transition, cleanup, and pre-handshake frame size. Done means incomplete connections are bounded and timed out while successful handshakes still enter normal connection management.