S3 connection pool exhausted over hours after enabling DROP_WITH_PURGE_ENABLED (1.7.0)

### Describe the bug

On Polaris 1.7.0 the S3 HTTP connection pool becomes progressively exhausted
until the server can no longer answer `/q/health` within a probe timeout and the
container is killed. Recovery is only by restart.

Time-to-failure scales with the configured pool size, which points at
connections being leaked per request rather than the pool being undersized.
Setting `polaris.storage.connection-time-to-live` and `connection-max-idle-time`
does **not** bound it — which narrows the leak to connections held as live
leases rather than left stale in the pool, since both settings are documented as
applying to a *pooled* connection.

Onset correlates exactly with enabling `DROP_WITH_PURGE_ENABLED`.

**Environment**

| | |
|---|---|
| Polaris | 1.7.0 (`apache/polaris:1.7.0`) |
| Runtime | Quarkus 3.37.4, JVM, `MaxRAMPercentage=80` |
| Persistence | `relational-jdbc` (PostgreSQL) |
| Object store | Ceph RGW, S3-compatible, **no STS endpoint** |
| Credentials | one static identity via the default AWS provider chain; no credential vending |
| Deployment | Kubernetes, 1 then 3 replicas, container limit 8 GiB |
| Clients | Trino (Iceberg REST connector) and Spark; workload is dbt, heavy on `table` materialisation |

### To Reproduce

1. Deploy Polaris 1.7.0 with `polaris.persistence.type=relational-jdbc` against an
S3-compatible store that has **no STS endpoint**, so Polaris uses one static
credential from the default AWS provider chain rather than vending per-catalog
credentials.
2. Set `polaris.features.DROP_WITH_PURGE_ENABLED=true`. This is required for
Trino, whose Iceberg REST connector always sends `DROP TABLE` with
`purgeRequested=true`; with the default `false` every drop returns
`403 Unable to purge entity`.
3. Point Trino at the catalog and run a sustained dbt workload using `table`
materialisation. Each model run ends by dropping `__dbt_backup`, so
purge is exercised continuously. Ours is roughly 500 requests/minute across
three replicas.
4. Watch `ConnectionPoolTimeoutException` accrue in the server log. They begin
long before any client impact.
5. Wait. Time-to-failure depends on `polaris.storage.max-http-connections`:
roughly 6 hours at the default, roughly 20 hours at 500. Eventually
`/q/health` cannot answer inside the probe timeout and the orchestrator kills
the container.

Restarting the pod resets it completely and the cycle repeats, which is what
makes it look like a leak rather than a sizing problem.

### Actual Behavior

The pool exhausts and every operation that needs S3 fails:

```
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(...)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionRequestFactory$InstrumentedConnectionRequest.get(...)
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:261)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(...)
```

Surfacing to clients as HTTP 500:

```
Unhandled exception returning INTERNAL_SERVER_ERROR:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request
```

on the endpoints where Polaris itself reads or writes S3 — `loadTable`, which
reads the table's `metadata.json`, and commits:

```
POST /api/catalog/v1/{catalog}/namespaces/{ns}/tables/{table} -> 500
GET /api/catalog/v1/{catalog}/namespaces/{ns}/tables/{table} -> 500
```

Downstream this reads as `ICEBERG_CATALOG_ERROR: Failed to load table: ` on
tables that are perfectly healthy, and `ICEBERG_COMMIT_ERROR` on writes. It is
intermittent, so a retry usually succeeds.

**Time-to-failure scales with pool size**

| `max-http-connections` | replicas | limit | time to failure |
|---|---|---|---|
| unset (default) | 1 | 4 GiB | **5 h 41 m** |
| 500 | 3 | 8 GiB | **20 h 17 m** |

It never stabilised. If 500 were simply the right size, survival would be
unbounded rather than proportionally longer.

**It concentrates on one replica at a time**

Three replicas behind one Service; exactly one degrades while the others stay at
zero, at the same age and near-identical request volume (~10,000 requests each
over the window):

```
pod A 66 min old 938 pool timeouts 10x HTTP 500 30x SdkClientException
pod B 65 min old 0 0 0
pod C 65 min old 0 0 0
```

This has happened three times, on a different pod each time, so it is not a bad
node or host.

**Memory grows in step, but is not the cause of death**

A pod starts near 500 MiB and climbs monotonically — 4,238 MiB at 21 hours,
about 180 MiB/hour. Consistent with leaked connection objects retaining buffers.
It is not OOM: no `OutOfMemoryError`, no GC-overhead error, no `OOMKilled`
event, and the container limit is 8 GiB.

**The kill is a health-probe timeout**

```
Warning Unhealthy pod/polaris-... Readiness probe failed
Warning Unhealthy pod/polaris-... Liveness probe failed
```

Exit 137, reason `Error`, with 273 pool-acquisition timeouts in the final 5.5
minutes. Request threads block for up to
`polaris.storage.connection-acquisition-timeout` waiting for a connection, so
the health endpoint cannot answer in time.

### Expected Behavior

S3 connections should be returned to the pool after every operation, including
the purge / file-cleanup path and all error and timeout paths, so that pool
utilisation is stable under sustained load rather than growing monotonically
until the server is unresponsive.

Concretely:

- A long-running Polaris instance under continuous `DROP TABLE ... PURGE` load should reach a steady state, not exhaust its pool.
- If `polaris.storage.connection-time-to-live` and `connection-max-idle-time` are **not** expected to reclaim connections that are leased but never returned, that limitation should be documented — we deployed them expecting them to bound this, and they did not.

### Additional context

**Onset correlates with `DROP_WITH_PURGE_ENABLED` — our strongest lead**

| | state | observed |
|---|---|---|
| day 1 | `DROP_WITH_PURGE_ENABLED=false` (default) | drops refused with `403 Unable to purge entity` — **no pool timeouts at all** |
| day 2 | `DROP_WITH_PURGE_ENABLED=true` | first pool exhaustion: **573 timeouts in 5.5 minutes** at pool 200 |

Same workload, same catalogs, same credential. Enabling purge turned
`DROP TABLE` from a metadata operation into a bulk object delete — one dropped
table can be thousands of `DeleteObject` calls — and the exhaustion started that
day.

Alongside the timeouts we also see failed deletes, which suggests the delete
path is where connections are lost:

```
Error encountered attempting to delete file (178 occurrences in one window)
org.apache.http.NoHttpResponseException: The target server failed to respond
```

**Negative result: connection lifetime does not bound it**

```
polaris.storage.connection-time-to-live = PT5M
polaris.storage.connection-max-idle-time = PT1M
```

Within 66 minutes of that deploy, one pod had 938 pool timeouts, 10 HTTP 500s
and 30 `SdkClientException`s while its two siblings had zero of each. Reverted.

We think this is informative rather than just a failed attempt: both settings
apply to a **pooled** connection, and a connection checked out and never
returned is *leased*, not pooled, so the eviction sweep never sees it. That
places the leak in the borrow/return path.

**Hypothesis (unverified — we have not read the source)**

Something in the purge / file-cleanup path acquires connections and does not
release them, most likely a response body left unclosed on a non-happy path, or
an async task abandoned before releasing its connection.
`polaris.features."POLARIS_TASK_TIMEOUT_MILLIS"` defaults to 300000 ms; a purge
task abandoned at that timeout mid-delete would plausibly leave in-flight
connections leased indefinitely.

**What we tried**

| change | result |
|---|---|
| `max-http-connections` default → 200 | client errors gone for a few hours, then returned |
| `max-http-connections` 200 → 500 | survival 5h41m → 20h17m; leak persists |
| 1 → 3 replicas | dilutes it threefold; one pod still collapses at a time |
| `connection-time-to-live` PT5M + `max-idle-time` PT1M | no improvement; reverted |
| liveness probe timeout 1s → 5s | stops premature kills of a merely-slow pod; the pod still dies once the pool is gone |
| periodic rolling restart | current mitigation, obviously a workaround |

**Impact**

- Polaris kills itself every 6–20 hours depending on pool size, requiring a restart to recover.
- Intermittent `ICEBERG_CATALOG_ERROR` / `ICEBERG_COMMIT_ERROR` on healthy tables, which downstream data teams cannot distinguish from their own bugs. Several days were spent investigating query logic before we identified the platform cause.
- Cascading failures in orchestrated pipelines: one blocked model stops every dependent model.

**What would help**

1. Confirmation of whether the purge / task-executor path releases S3 connections on all paths, including timeout and error.
2. Whether `polaris.storage.*` lifetime settings are expected to reclaim leased connections at all.
3. A connection-pool metric on `/q/metrics` for leased vs available connections. We could not find one; it would have made this diagnosable in minutes rather than days.

Happy to run diagnostics, apply a patch, or capture a heap dump from a degraded
pod — we can reproduce this reliably in under a day.

### System information

_No response_

Start by tracing the DROP TABLE purge/file-cleanup path and its S3 DeleteObject calls, including response handling and task timeouts. Reproduce the sustained workload and observe /q/health, /q/metrics, and pool timeout logs. Done means connections are returned on success, error, and timeout paths so pool usage stabilizes without restarts.

Written by the indexing model from the issue text.