Thread Leak: OOM After 24h Operation (1,309 Threads)

# Thread Leak: OOM After 24h Operation (1,309 Threads)

## Problem

Production workers running Kubernetes plugin tasks experience unbounded thread growth, leading to OOM after ~24 hours.

**Observed symptoms:**
- Thread count: 1,309 (261% over configured 500-thread limit)
- 168 leaked threads in 89 orphaned executor pools (pattern: `-2073792911-pool-*`)
- Threads stuck in TIMED_WAITING, never terminated
- Memory exhaustion despite adequate heap space (1.97GB used of 4GB)

**Thread dump evidence:**
```
"-2073792911-pool-734325-thread-1" daemon TIMED_WAITING
at java.util.concurrent.SynchronousQueue.poll()
at java.util.concurrent.ThreadPoolExecutor.getTask()
at java.util.concurrent.ThreadPoolExecutor.runWorker()
```

## Affected Task

**Only `PodCreate`** - all kubectl tasks (`Apply`, `Get`, `Delete`, `Patch`, `Restart`) are unaffected.

## Environment

- Kestra v1.1.3
- Plugin-kubernetes (current version)
- ~1,000,000 executions daily
- EKS v1.33.5, workers: 4 CPU / 8 GB RAM

## Analysis

### Try-With-Resources Pattern (Appears Correct)

[`PodCreate.java:355-356`](https://github.com/kestra-io/plugin-kubernetes/blob/main/src/main/java/io/kestra/plugin/kubernetes/core/PodCreate.java#L355-L356):
```java
try (KubernetesClient client = PodService.client(runContext, this.getConnection());
PodLogService podLogService = new PodLogService(...)) {
// operations
}
```

Resources close in reverse order: `podLogService.close()`, then `client.close()`.

### Suspect: Async Executor Not Fully Terminated

**Theory (to be confirmed):**

[`PodLogService.java:196-198`](https://github.com/kestra-io/plugin-kubernetes/blob/main/src/main/java/io/kestra/plugin/kubernetes/services/PodLogService.java#L196-L198):
```java
if (scheduledExecutor != null) {
scheduledExecutor.shutdownNow();
// Returns immediately - executor thread may still be running
}
```

The scheduled executor (runs every 30s) captures `KubernetesClient` in its lambda closure:

[`PodLogService.java:64`](https://github.com/kestra-io/plugin-kubernetes/blob/main/src/main/java/io/kestra/plugin/kubernetes/services/PodLogService.java#L64):
```java
scheduledExecutor.scheduleAtFixedRate(
() -> {
PodResource podResource = PodService.podRef(client, pod); // Captures client
// ... creates watchLog() connections
},
0, 30, TimeUnit.SECONDS
);
```

**Suspected race condition:**
1. `podLogService.close()` calls `shutdownNow()` and returns immediately
2. Scheduled thread still running, holds reference to `client`
3. `client.close()` attempts to close internal OkHttp pools
4. Pools cannot fully close (reference still held)
5. OkHttp threads leak

**Missing:** `scheduledExecutor.awaitTermination(timeout, unit)` after `shutdownNow()`

### Why Hard to Detect

- Timing-dependent (race between cleanup and scheduled task)
- Small leak per operation (~2 threads)
- Daemon threads don't prevent JVM exit (unit tests pass)
- Only manifests under sustained load over 24+ hours

## Proposal

I suggest investigating whether:
1. The scheduled executor in `PodLogService` fully terminates before `client.close()` is called
2. Missing `awaitTermination()` is the root cause
3. Other async operations might hold client references

## Related

- [kestra#13129](https://github.com/kestra-io/kestra/issues/13129) - Original OOM report
- [Full analysis](https://gist.github.com/cf-sewe/6c3597cb03749ed77a5c7f36db801269) - Detailed investigation

Start with PodLogService.java:64 and 196-198, then inspect PodCreate.java:355-356 to trace executor and client shutdown ordering. Reproduce repeated PodCreate operations under sustained load and observe executor termination, orphaned pools, and thread counts. Done means the root cause is confirmed or ruled out with evidence and the cleanup behavior is verified without leaked threads.

Written by the indexing model from the issue text.