[Perf] Investigate tokenizer performance under high concurrency for prefix cache routing

### 🚀 Feature Description and Motivation

Tokenization sits in the hot path of prefix cache-aware routing — every request must be tokenized before `MatchPrefix()` can run. Under high concurrency with long prompts, this can become a CPU/latency bottleneck.

Areas to investigate:
- CPU resource limits: In K8s, the gateway plugin pod's CPU request/limit directly caps tokenization throughput. Need benchmarks: measure tokenization latency × QPS for tiktoken vs remote at various prompt lengths
(1K, 4K, 16K, 64K tokens) to establish CPU sizing guidance.

- AIBrix currently does not use HuggingFace AutoTokenizer — it uses tiktoken (local, Go) or remote tokenization via HTTP to vLLM/SGLang endpoints. We need to support local tokenizer. technically, we can add the dynamic load feature to read the models

- TensorRT-LLM compatibility: `tiktoken` produces different token IDs than TRT-LLM's tokenizer (SentencePiece/HF-based). For TRT-LLM prefix cache-aware routing, we either need to call TRT-LLM's /tokenize endpoint(remote) or load the model's HF tokenizer locally. Neither path exists today. Note:

### Use Case

optimization for prefix-cache and performance guidance for gateway

### Proposed Solution

_No response_