Change the batch job management to application master way

### 🚀 Feature Description and Motivation

I'd like to borrow the idea from YARN to solve some problems in batch jobs.

- Current: Metadata service orchestrates everything. Worker Job = dumb executor, just sends requests to a given endpoint
- YARN AM: Worker Job is the intelligent coordinator. Metadata Service manage control plan. Worker Job is responsible for spawning job workers. and then execute batch requests. resource negotiation and gpu worker lifecycle could be managed by Job API.

## Key Advantages

1. Decoupled lifecycle: AM starts immediately (CPU is cheap/fast). Resource acquisition happens asynchronously. No need for metadata service to block on "wait for Deployment/StormService ready."
2. AM has local state: The AM can track which requests succeeded/failed, manage retries, adjust concurrency — without constant round-trips to metadata service. Current architecture stores all state in Redis/annotations.
4. Clean preemption: When preempted, the AM saves a checkpoint (which requests completed), releases resources via RM API, and exits. When restarted, it picks up from the checkpoint.
5. Observable: AM exposes its own /status endpoint. Console/monitoring can scrape it directly. No need to reconstruct state from K8s annotations, we probably can use DB to sync status as well.

```
┌───────────────────────────┬───────────────────────────────────┬────────────────────────────────────────────┐
│ Aspect │ Before │ After (YARN AM) │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Who creates StormService? │ Metadata service │ Metadata service (RM), but requested by AM │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Who waits for ready? │ Metadata service │ AM (worker Job) │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Who manages progress? │ Metadata service via kopf + Redis │ AM locally + reports to RM │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Who handles preemption? │ Metadata service signals worker │ RM signals AM, AM checkpoints │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Worker Job is... │ Dumb request executor │ Intelligent job coordinator │
├───────────────────────────┼───────────────────────────────────┼────────────────────────────────────────────┤
│ Metadata service is... │ Orchestrator + executor │ Resource Manager + API surface │
└───────────────────────────┴───────────────────────────────────┴────────────────────────────────────────────┘
```

### Use Case

better architecture design for batch users.

### Proposed Solution

_No response_

Start by mapping the current metadata service, Worker Job, Redis or Kubernetes-annotation state, and Job API interactions described in the issue. Review how StormService readiness, progress, preemption, and retries are handled today. Done requires an agreed application-master architecture and an implementation plan with defined lifecycle, checkpointing, status, and resource-management behavior.

Written by the indexing model from the issue text.