Queries may be blocked forever when poco session pool is enabled under disagg arch

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

In a previous commit, I introduce storage.s3.enable_http_pool and enable it by default. It help us reduce some error when testing with MinIO at that time. But I find it make queries hang in the latest version.

1. Deploy a tidb cluster with 1 wn and 1 rn (disagg arch)
2. Load chbenchmark with 1 tiflash replica
3. Execute set tidb_isolation_read_engines = 'tiflash'; select count(*) from order_line; and select * from order_line limit 1;

2. What did you expect to see? (Required)

All queries finish normally

3. What did you see instead (Required)

The last query get blocks, and we can see following message in the logging

[2023/06/06 16:37:37.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:38.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:39.906 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:40.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:41.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:42.907 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:43.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:44.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:45.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:46.908 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:47.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:48.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:49.909 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:50.910 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]
[2023/06/06 16:37:51.910 +08:00] [INFO] [PoolBase.h:152] ["No free connections in pool. Waiting 1000000 ms."] [source=HTTPSessionPool] [thread_id=108]

4. What is your TiFlash version? (Required)

https://github.com/pingcap/tiflash/commit/edf82d2a63c76bf87b15efc18fbac3c0d195ddae

Start by reproducing the disaggregated TiFlash setup and the two order_line queries with storage.s3.enable_http_pool enabled. Inspect PoolBase.h and the HTTPSessionPool log path to determine why the pool has no available connections. Done means both queries complete normally without repeated pool-wait messages.