BE remote scanner thread pool (rs_normal) never shrinks — thread count grows to 27k+ and BE freezes

# BE remote scanner thread pool (`rs_normal`) never shrinks — thread count grows to 27k+ and BE freezes

## Environment

- **Doris version**: 4.1.1-rc01
- **Cluster size**: 4 BE + 3 FE, deployed via DorisOperator on Kubernetes
- **BE image**: official `apache/doris:doris-4.1.1-rc01-be` (x86_64)
- **Workload**: BE queries external tables backed by Paimon on S3 (Doris × Paimon catalog), mixed with normal internal OLAP queries
- **BE resources**: cpu 8 core, memory limit 40G (cgroup), `mem_limit=32G`

## Symptom

BE pod RSS climbs monotonically from ~3G to 6–7G over ~4h, then either:
- keeps growing until the pod is OOM-ish and gets killed by cgroup, **or**
- at ~27k threads the BE process freezes: it stops responding to FE heartbeats, `SHOW BACKENDS` shows `Alive=false` with `SocketTimeoutException: Read timed out`, CPU drops to ~1m, readiness probe times out. **Not** OOMKilled — internal lock/resource exhaustion.

The only recovery is `kubectl delete pod` (restart the BE process).

## Misleading signals

All conventional memory metrics stay green, which made this look like a "memory leak" for a while:

- `JVM heap used` — normal GC sawtooth, returns to baseline
- `jemalloc resident` — stable at ~2G
- `jvm non-heap` — stable at ~130M
- `mem_tracker` — stable

## Root cause

`/proc/$BE_PID/status` shows `Threads=16000–27000`, of which ~90% are `rs_normal [worker]` threads in `S` (sleeping) state. Each thread has a 512K stack → **7–12GB just for thread stacks**, which is what pushes RSS up.

Thread census command:
```bash
BE_PID=$(pgrep -f doris_be)
ls /proc/$BE_PID/task/ | while read t; do cat /proc/$BE_PID/task/$t/comm; done \
| sort | uniq -c | sort -rn | head
```

Typical output on a frozen BE:
```
27012 rs_normal
48 olap_scan
24 brpc
...
```

The `rs_normal` pool (remote scanner, used for querying external catalogs like Paimon/Hive/Iceberg) **only grows, never shrinks**. Idle keep-alive eviction either isn't implemented or isn't effective in 4.1.1-rc01.

## Configs I tried to cap the pool (all ineffective on rc01)

| # | Where | Setting | Result |
|---|-------|---------|--------|
| 1 | `be.conf` | `doris_scanner_thread_pool_thread_num=256` | Only caps `ls_normal` (local scanner). `rs_normal` still linear-grows. |
| 2 | `be.conf` | `doris_max_remote_scanner_thread_pool_thread_num=256` | **No effect.** `rs_normal` keeps growing past 256. |
| 3 | FE | `ALTER WORKLOAD GROUP normal PROPERTIES ('max_remote_scan_thread_num'='256', 'min_remote_scan_thread_num'='48');` | **No effect.** |

Related defaults on this BE:
- `doris_remote_scanner_thread_pool_thread_num=48` (initial)
- `doris_max_remote_scanner_thread_pool_thread_num=-1` (unlimited)

## Current workaround

CronJob that rotates one BE pod every 4h (`kubectl delete pod` on the oldest ready BE, with guards: `ready BE >= 3` and `pod age > 6h`). This keeps the leak from ever reaching the freeze threshold, but it's purely symptomatic.

YAML for the workaround is available if useful.

## Expected behavior

At least one of these should actually work on rc01:

1. `doris_max_remote_scanner_thread_pool_thread_num=N` should hard-cap `rs_normal` size, excess idle threads should be reaped when the cap is hit and load drops.
2. Idle `rs_normal` workers should be evicted after a keep-alive timeout (like `ls_normal` does), so the pool shrinks back to `doris_remote_scanner_thread_pool_thread_num` when external-scan load stops.
3. `ALTER WORKLOAD GROUP … max_remote_scan_thread_num=N` should bound concurrent remote scans per workload group — currently appears to be ignored or not enforced against the `rs_normal` pool.

## Reproduction

1. Fresh BE 4.1.1-rc01 with a Paimon catalog on S3.
2. Run a sustained mixed workload that issues external scans against Paimon (a few QPS is enough; the pool grows even under modest load).
3. Within 4–8h, `Threads=` in `/proc/$BE_PID/status` crosses 20k and RSS climbs correspondingly.
4. Eventually (usually 24k–28k threads) the BE stops answering FE heartbeats and has to be killed externally.

Happy to provide thread dumps, `be.conf`, `SHOW BACKENDS`/`SHOW WORKLOAD GROUPS` output, or a heap profile if it helps. Just let me know what's most useful.

## Question

Is this a known issue in 4.1.1-rc01? Is there a fix in a later 4.1.x or 4.2.x release? If a fix exists, I'd like to upgrade instead of running the rotator CronJob forever.

Start with the rs_normal thread census using /proc/$BE_PID/status and the provided worker-count command, then compare behavior with the be.conf remote scanner settings and ALTER WORKLOAD GROUP limits. Reproduce the sustained Paimon-on-S3 workload and verify that rs_normal is capped or reaps idle workers, RSS stops growing, and the BE continues answering FE heartbeats.

Written by the indexing model from the issue text.