Improve load balancing and TCP connection TTL handling in Logstash outputs

# Improve load balancing and TCP connection TTL handling in Logstash outputs

## Summary

Logstash output plugins (most notably `logstash-output-elasticsearch`, but this also affects
other TCP/HTTP-based outputs) lack robust load-balancing semantics and configurable TCP
connection TTL / rotation.

In real-world HA deployments this leads to uneven traffic distribution, sticky connections,
slow failover, and operational pain when downstream endpoints are restarted, rotated, or
fronted by load balancers.

---

## Problem Description

### 1. Load balancing is connection-sticky, not event-aware

When multiple hosts are configured in an output, for example:

```ruby
hosts => [
"es-node-1:9200",
"es-node-2:9200",
"es-node-3:9200"
]
```

Logstash establishes persistent connections and then:

- Pins pipeline workers to a subset of hosts
- Distributes events **per connection**, not per event or per batch
- Frequently results in **hot nodes and cold nodes**

Even with multiple pipeline workers, traffic distribution is often uneven and difficult to
reason about, especially under bursty workloads or uneven pipelines.

---

### 2. No configurable TCP / HTTP connection TTL

Once a connection is established, it may live indefinitely unless:

- The remote end forcibly closes it
- Logstash is restarted
- A hard failure occurs

There is currently no supported way to express:

> “Recycle output connections every _N_ seconds / _N_ events.”

This causes issues when:

- Load balancers rely on connection churn
- Backends are rotated (cloud scaling, blue/green, AZ failover)
- Certificates or routes change
- Downstream Logstash or Elasticsearch nodes are restarted

Connections can remain pinned for long periods, defeating expected load-balancer behavior.

---

### 3. Slow or uneven failover

When a downstream endpoint becomes unhealthy:

- Existing connections may continue to be reused
- Failures can cluster on a single pipeline worker
- Recovery can be slow and uneven without restarting Logstash

This is particularly visible in Elastic Cloud or managed load-balancer environments.

---

## Expected / Desired Behavior

### A. Improved load-balancing strategies

Examples of desirable strategies (implementation-agnostic):

- Round-robin per batch
- Round-robin per event
- Least-connections
- Hash-based routing (event-field aware)

Even a simple, deterministic **whole-batch round-robin across hosts** would be a significant
improvement over current behavior.

---

### B. Configurable connection TTL / rotation

Illustrative configuration options (names not prescriptive):

```ruby
connection_ttl => 300s
max_events_per_connection => 1000000
max_batches_per_connection => 10000
```

This would enable:

- Predictable connection churn
- Fair redistribution across hosts
- Better compatibility with modern L4/L7 load balancers
- Safer certificate, route, and backend changes without restarts

---

### C. Faster and cleaner failover

- Proactive eviction of connections after repeated failures
- Faster redistribution of traffic when endpoints recover
- Reduced operational need to restart Logstash to rebalance outputs

---

## Why This Matters

These limitations:

- Impact high-availability and horizontal scalability
- Complicate cloud and load-balancer-based deployments
- Force operators to add external proxies or restart Logstash as a workaround
- Become more severe at scale (multiple pipelines, high event throughput)

Modern ingest architectures expect connection lifecycle control as a first-class feature.

---

## Environment (Example)

- Logstash: 8.x / 9.x
- Output plugin: `logstash-output-elasticsearch`
- Deployment: multi-node AWS Logstash to Elastic Cloud / LB-fronted Elasticsearch
- Pipelines: multiple high-throughput pipelines sharing outputs
- Load Balancer: AWS Network Load Balancer distributing logstash connections between a large pool of Logstash servers.

---

# Logstash to Logstash Meta reference

"Load-balancing: an output should be configurable with hosts => ["12.34.56.78:1234", "12.34.56.79:1234"] or hosts => ["12.34.56.78", "12.34.56.79"] port => 1234 (alternatives to host => "12.34.56.78" port => 1234) and should distribute events to the downstream hosts (naive whole-batch round-robin is ok at first, but then we could make it better)
SSL: fill gaps in identity and trust configuration."

https://github.com/elastic/logstash/issues/15163