Extend Async KV Requests to More Execution Paths

## Enhancement

**Goroutine is good, but it is not free**.

## Summary

Following the successful implementation of async KV requests for batch get operations ([client-go#1586](https://github.com/tikv/client-go/issues/1586), [tidb#64826](https://github.com/pingcap/tidb/pull/64826)), this issue proposes extending the async KV request pattern to additional execution paths to reduce goroutine overhead and improve overall system performance.

## Background

### Current Implementation

The async KV request infrastructure in client-go provides:

- **Core Interfaces** (`util/async/core.go`):
- `Pool` - Base interface for goroutine pool implementations
- `Executor` - Extends Pool with safe concurrent function batching via `Append()`
- `Callback[T]` - Generic callback interface supporting deferred transformations, immediate invocation, and scheduled execution

- **RunLoop** (`util/async/runloop.go`):
- Single-goroutine execution model with state machine (Idle/Waiting/Running)
- Efficient task batching via slice swaps
- Context-aware cancellation support

- **Async RPC Interface** (`internal/client/client_async.go`):
- `SendRequestAsync()` method on `RPCClient`
- Leverages existing batch client's send loop and recv loop mechanisms
- Priority queue for request ordering

This architecture **eliminates the need to spawn a new goroutine for each concurrent KV request by using callbacks and a centralized execution loop**.

### Benefits Achieved with Batch Get

The batch get implementation demonstrated:
- Reduced goroutine creation overhead
- Lower memory pressure from goroutine stacks
- Better CPU cache utilization
- Maintained concurrent request capability without excessive parallelism

### 1. 2PC Prewrite Phase

**Current Behavior:**

The 2PC prewrite phase (controlled by `tidb_committer_concurrency`, default: 128) spawns one goroutine per region batch:

```go
// client-go/txnkv/transaction/2pc.go:2241
go func() {
defer batchExe.rateLimiter.PutToken()
// ... setup backoffer
ch <- batchExe.action.handleSingleBatch(batchExe.committer, singleBatchBackoffer, batch)
}()
```

**Key Files:**
- `client-go/txnkv/transaction/2pc.go` - `batchExecutor.startWorker()` (line 2235)
- `client-go/txnkv/transaction/prewrite.go` - `actionPrewrite.handleSingleBatch()` (line 222)
- `client-go/util/rate_limit.go` - Token-based rate limiter

**Proposed Change:**

Replace the goroutine-per-batch pattern with async KV requests:
1. Use `SendReqAsync()` for prewrite requests instead of blocking `SendReq()`
2. Use `RunLoop` to execute callbacks from multiple batches in a single goroutine
3. Maintain the `tidb_committer_concurrency` semantic as a limit on concurrent in-flight requests

**Expected Benefits:**
- Large transactions with many region batches will see significant goroutine reduction
- Commit latency improvement for write-heavy workloads
- Better resource utilization during peak commit phases

In production, we have seen high tail latency caused by committer token wait as the related transaction involves thousands of regions. And it is quite inconvenient for user to set this systerm variable properly.

### 2. Coprocessor and DistSQL

**Current Behavior:**

The coprocessor implementation spawns a fixed worker pool at query start:

```go
// tidb/pkg/store/copr/coprocessor.go:1088-1094
for i := range it.concurrency + it.smallTaskConcurrency {
ch := taskCh
if i >= it.concurrency && smallTaskCh != nil {
ch = smallTaskCh
}
worker := newCopIteratorWorker(it, ch)
go worker.run(ctx) // <-- Goroutine spawned here
}
```

**Concurrency Parameters:**
| Parameter | Default | Description |
|-----------|---------|-------------|
| `tidb_distsql_scan_concurrency` | 15 | General distributed SQL scan |
| `tidb_analyze_distsql_scan_concurrency` | 4 | Analyze statement scans |
| `tidb_index_lookup_concurrency` | ConcurrencyUnset | Index lookup operations |

**Key Files:**
- `tidb/pkg/store/copr/coprocessor.go` - `copIterator.open()` (line 1070), `copIteratorWorker.run()`
- `tidb/pkg/store/copr/batch_coprocessor.go` - `batchCopIterator.run()` (line 1301)
- `tidb/pkg/distsql/request_builder.go` - Concurrency configuration (line 94)

**Proposed Change:**

1. Refactor `copIteratorWorker` to use async KV requests:
- Replace blocking `SendKVReq()` with `SendReqAsync()`
- Use `RunLoop` pattern for callback execution
- Reduce worker pool size while maintaining throughput

2. For batch coprocessor (TiFlash):
- Similar refactoring for `batchCopIterator.handleTask()`

**Expected Benefits:**
- Reduced memory footprint for concurrent queries
- Better scheduling efficiency for mixed workloads
- Consistent latency under high query concurrency

### 3. Other Potential Modules

Additional execution paths that could benefit from async KV requests:

| Module | Current Pattern | Files |
|--------|-----------------|-------|
| Index Lookup Executor | Worker pool per lookup batch | `tidb/pkg/executor/index_lookup*.go` |
| Point Get Executor | Direct sync calls | `tidb/pkg/executor/point_get.go` |
| Batch Point Get | Goroutine per batch | `tidb/pkg/executor/batch_point_get.go` |
| 2PC Commit Phase | Similar to prewrite | `client-go/txnkv/transaction/commit.go` |
| Pessimistic Lock | Goroutine per batch | `client-go/txnkv/transaction/pessimistic.go` |
| Resolve Lock | Concurrent resolution | `client-go/txnkv/txnlock/lock_resolver.go` |

## Implementation Plan

### Phase 1: 2PC Prewrite (High Priority)

- [ ] **Task 1.1:** Add `SendReqAsync` support in `actionPrewrite.handleSingleBatch()`
- [ ] **Task 1.2:** Refactor `batchExecutor` to use `RunLoop` instead of spawning goroutines
- [ ] **Task 1.3:** Ensure `tidb_committer_concurrency` still controls max concurrent requests
- [ ] **Task 1.4:** Add unit tests and integration tests
- [ ] **Task 1.5:** Benchmark commit latency for large transactions

### Phase 2: Coprocessor/DistSQL (Medium Priority)

- [ ] **Task 2.1:** Refactor `copIteratorWorker` to use async requests
- [ ] **Task 2.2:** Update `copIterator.open()` to use `RunLoop` pattern
- [ ] **Task 2.3:** Refactor `batchCopIterator.handleTask()` for TiFlash
- [ ] **Task 2.4:** Ensure `tidb_distsql_scan_concurrency` semantics preserved
- [ ] **Task 2.5:** Add comprehensive tests for streaming results
- [ ] **Task 2.6:** Benchmark query latency and throughput

### Phase 3: Other Executors (Lower Priority)

- [ ] **Task 3.1:** Identify additional executors that would benefit
- [ ] **Task 3.2:** Prioritize based on usage patterns and potential impact
- [ ] **Task 3.3:** Implement async patterns for selected executors
- [ ] **Task 3.4:** Ensure backward compatibility with existing configurations

## Compatibility Considerations

1. **Session Variables:** All existing concurrency-related session variables should continue to work as limits on concurrent in-flight requests.

2. **Error Handling:** The async callback pattern must properly propagate errors and support context cancellation.

3. **Metrics:** Existing latency and throughput metrics should remain functional with the new implementation.

4. **Pipelined DML:** The existing pipelined DML feature (`tidb_enable_pipelined_dml`) uses its own async patterns and should be evaluated for potential integration.

## References

- Async KV Request Infrastructure: [client-go#1586](https://github.com/tikv/client-go/issues/1586)
- Batch Get Implementation:
- [client-go#1591](https://github.com/tikv/client-go/pull/1591) - Core interfaces
- [client-go#1604](https://github.com/tikv/client-go/pull/1604) - RPCClient async support
- [client-go#1618](https://github.com/tikv/client-go/pull/1618) - RegionRequestSender async
- [client-go#1646](https://github.com/tikv/client-go/pull/1646) - KVSnapshot async batch-get
- [tidb#64826](https://github.com/pingcap/tidb/pull/64826) - TiDB integration