Control-plane polling churns connections at the idle boundary and across more than five Apache routes

Summary

The separate /localnodes discovery and health-probe transport has two built-in connection-churn mechanisms:

1. Its AWS SDK idle timeout is 60 seconds, exactly matching the default idle refresh interval.
2. Its Apache pool has a global capacity of five by default, while cluster-scope discovery contacts every discovered endpoint.

This follows up on the connection-reuse work tracked by #10.

Idle-boundary churn

ApacheSyncClientFactory.createPollingClient and CrtSyncClientFactory.createPollingClient configure TCP keepalive and pool capacity, but do not apply AlternatorConfig.connectionMaxIdleTimeMs. They therefore inherit the AWS SDK 2.42.2 default idle limit of 60 seconds.

AlternatorLiveNodes schedules the next idle refresh 60 seconds after the previous refresh finishes. A released polling connection is therefore at least 60 seconds idle when the next refresh leases it. Apache closes the expired entry and opens a replacement; CRT has the same 60-second native idle limit.

Relevant code:

• src/main/java/com/scylladb/alternator/internal/ApacheSyncClientFactory.java:98-108
• src/main/java/com/scylladb/alternator/internal/CrtSyncClientFactory.java:109-124
• src/main/java/com/scylladb/alternator/AlternatorConfig.java:71-78
• src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java:90-145, 232-249

A raw-socket probe confirmed that an Apache polling client used a new client port after 65 seconds, while a main client configured with the 600-second default reused its original port.

Apache route-capacity churn

The polling pool is sized to healthProbeConcurrency + 1, which is five by default. Apache applies this as both its global and per-route maximum.

Default ClusterScope discovery iterates all discovered candidates. Once more than five endpoint routes are used, Apache must evict an idle LRU route before allocating another. Candidate order is shuffled each refresh, so larger clusters can repeatedly close and reopen polling connections. During recent application traffic, refreshes can run every second.

Known-down nodes are also included after active and quarantined candidates, creating repeated connection attempts in cluster scope in addition to the 30-second health probes.

Relevant code:

• src/main/java/com/scylladb/alternator/AlternatorDynamoDbClient.java:816-835
• src/main/java/com/scylladb/alternator/AlternatorDynamoDbAsyncClient.java:799-806
• src/main/java/com/scylladb/alternator/NodeHealthConfig.java:34-41
• src/main/java/com/scylladb/alternator/internal/AlternatorLiveNodes.java:683-778

Impact

• Periodic TCP/TLS handshakes in otherwise idle applications.
• High connection churn in clusters with more than five nodes when Apache is selected for polling.
• Additional failed connection attempts to known-down nodes.
• Async clients are also affected because they use a separate synchronous polling transport.

Expected behavior

Healthy control-plane connections should remain reusable across normal refresh intervals and across all discovered routes without unnecessary eviction.

Acceptance criteria

• Polling idle timeout exceeds the configured idle refresh interval or is explicitly configurable.
• Common connection settings are applied to the polling transport, or dedicated polling settings are exposed.
• Apache polling capacity does not force route eviction solely because the cluster contains more than five nodes.
• Cluster-scope treatment of known-down nodes is reviewed to avoid excessive reconnect attempts.
• Socket-identity tests cover idle polling and clusters larger than the polling pool's previous global limit.

Start with ApacheSyncClientFactory.java, CrtSyncClientFactory.java, AlternatorConfig.java, and AlternatorLiveNodes.java to trace polling timeout, pool capacity, route selection, and refresh scheduling; then inspect the synchronous and asynchronous client call sites and NodeHealthConfig.java. Done means polling connections survive refresh intervals, routes are not evicted solely due to cluster size, known-down nodes do not cause excessive reconnects, and socket-identity tests cover idle polling and larger clusters.