[Umbrella][AI CLI] Roadmap Discussion: From Config Generator to Data Integration Copilot

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue) and found no similar issues.

### Description

## Background

The SeaTunnel AI CLI has completed its first phase. Shipped so far:

- Core capability: natural-language → SeaTunnel config generation, with a Planner → Generator → Validator → Repair multi-agent pipeline (#10789, #11109)
- Provider coverage: Anthropic / OpenAI-compatible / AWS Bedrock (compat fixes in #11509) / bedrock-mantle for Responses-API-only Bedrock models (#11548)
- Website documentation suite, en + zh (#11551, #11605)
- **Accuracy benchmark: 100 tasks, layered verdict gates up to real job execution, repair-loop measurement, 7 models evaluated (#11553; proposed and accepted in #11549; measured results on the website under AI CLI → Model Benchmark)**

With the benchmark in place, we now have *quantified* evidence of where generation fails and why — 13 tasks fail across nearly every model, and the failures cluster into four fixable knowledge gaps. This umbrella proposes a roadmap for what comes next, and invites the community to discuss scope and priorities — and to pick up sub-tasks.

## Vision

Evolve the AI CLI along three stages:

```
Stage 1: Generate accurately ── the current focus, backed by benchmark data
Stage 2: Operate reliably ── from generation tool to operations assistant
Stage 3: Broaden scenarios ── from tool to platform entry point
```

A guiding principle carried over from the benchmark work: **every accuracy-related change must be justified by benchmark deltas.** No prompt tweaks by intuition. The benchmark stamps the CLI version + commit into every report, so any change is A/B-comparable on identical tasks.

---

## Stage 1 — Generation Accuracy (active now)

Benchmark analysis identified four systematic knowledge gaps (13 tasks failed by 6–7 of 7 models) and two behavioral defects. Each item below is a small, independent PR.

| # | Item | Evidence | Status |
|---|---|---|---|
| 1.1 | Knowledge injection: Doris/StarRocks connector options (fenodes, load ports, save mode) | 3 tasks failed by 6–7/7 models | 🟢 ready, **help wanted** |
| 1.2 | Knowledge injection: conditional-routing golden example (one source → parallel SQL transforms → different sinks) | 3 tasks failed by 5–6/7; weakest models score 0% in this category | 🟢 ready, **help wanted** |
| 1.3 | Knowledge injection: PostgreSQL-CDC prerequisites (replication slot, publication, decoding plugin) | CDC category: 25–50% on weaker models | 🟢 ready, **help wanted** |
| 1.4 | Knowledge injection: wide-DAG composition rules (5+ blocks, label wiring) | 1 task failed by 5/7; Tier-3 drag | 🟢 ready, **help wanted** |
| 1.5 | `--no-ask` non-interactive mode: apply documented defaults instead of re-asking | over-clarification cost two models 10–14 tasks each | 🟢 ready |
| 1.6 | Connector-name fuzzy correction ("File" → "did you mean LocalFile?") in local validation | observed invented connector names | 🟢 ready, **good first issue** |
| 1.7 | Structured error parsing foundation (stacktrace → error code / component / parameters) | models repair structured errors at ~2× the rate of raw stack traces; only 47% of runtime failures recoverable today | design ready |
| 1.8 | Repair-prompt structured reflection for weak-repair models | repair recovery spread measured at +2 to +18 tasks across models | after 1.7 |
| 1.9 | **Benchmark as CI regression gate**: accuracy-affecting PRs report deltas on a smoke subset (10–15 L1 tasks, zero API cost in CI) | infrastructure | proposal |

Combined expectation for 1.1–1.4: **+8–13pp overall pass rate per model**, to be verified by before/after benchmark runs included in each PR.

### Proposed architecture direction: pluggable Skill Packs

Items 1.1–1.4 should **not** be hardcoded into prompts. Proposal: a declarative per-connector knowledge-pack format (loaded alongside the existing skills/golden-examples layers), so future connector knowledge can be contributed *without touching CLI core code* — analogous in effort to adding a connector doc page. This creates a low-barrier contribution channel; design sketch to follow in a sub-issue.

---

## Stage 2 — Runtime Operations (design discussion now; implementation after the Stage-1 foundation)

The engine already provides everything needed: diagnostic REST endpoints (`/job-info`, `/logs`, `/metrics`, `/thread-dump`, checkpoint history) and event push (`JobEventHttpReportHandler`). Proposed order:

| # | Item | Notes |
|---|---|---|
| 2.1 | **Security & permission model (RFC, prerequisite)** — read-only by default, explicit confirmation for mutating ops, audit log, confidence-tiered action whitelist | gate for everything below |
| 2.2 | `/diagnose `: pull job-info / logs / metrics / thread-dump → structured parsing (shared foundation with 1.7) → LLM root cause + suggested fix, wired into the existing repair agent | endpoints all available |
| 2.3 | `/status`, `/metrics`: deterministic tables first (zero LLM cost), optional LLM anomaly interpretation. **Explicit non-goal: replicating the Web UI** — the differentiation is conversational follow-up ("why is it slow?" → correlate thread dump + backpressure) | boundary needs community consensus (open question 1) |
| 2.4 | `/checkup`: whole-cluster health report (the category is validated by k8sgpt in the K8s ecosystem) | |
| 2.5 | Self-healing pipelines (daemon mode): engine event push → auto-diagnosis → confidence-tiered response (high confidence: auto-repair & resubmit with report; low: report & wait) | **strictly gated on 2.1**; highest narrative value, highest risk |

Diagnosis quality is measurable with the same benchmark methodology (inject known failures → score root-cause hit rate), keeping the "no intuition-driven AI features" principle.

---

## Stage 3 — Scenario Expansion (collecting feedback; not scheduled)

| # | Item | Positioning |
|---|---|---|
| 3.1 | `/explore`: Catalog API schema discovery → LLM recommends sync pipelines → one-click generation. Metadata-level only — no data sampling/profiling (privacy boundary) | inverts the interaction: from "you describe" to "I recommend"; richer real context is itself an accuracy lever |
| 3.2 | RAG pipeline entry ("AI configures AI"): LLM/embedding transforms and vector sinks already exist in SeaTunnel; the missing piece is a one-sentence entry point ("embed product descriptions into Milvus") with dimension/parameter validation | smallest effort in this stage |
| 3.3 | Schema drift negotiation: CDC `SchemaChangeEvent` → conversational DDL migration confirmation for the sink | real CDC user pain; type mapping is a measured model strength |
| 3.4 | `/migrate`: semantic understanding of legacy Sqoop/Kettle/bash/stored-proc jobs → equivalent SeaTunnel config + diff explanation | adoption-driven; complements x2seatunnel's rule-based conversion |
| 3.5 | `/simulate`: shadow-run sampled/synthetic data in a testcontainers micro-env; LLM explains behavioral diffs before production | the benchmark's real-execution gate, productized; heavy |
| 3.6 | NL data-quality sentinel: natural language → Assert rules + scheduled validation pipeline | opportunistic |
| 3.7 | MCP server mode: expose CLI capabilities (generate / diagnose / explore) to external agent frameworks | complements the existing seatunnel-mcp; tracking industry momentum |

---

## How to contribute

- Items marked **help wanted / good first issue** will get dedicated sub-issues with context and code pointers — comment here to claim one
- Once the Skill Pack format lands, connector-knowledge contribution requires **no Python/Java changes** — an ideal first contribution
- Every Stage-1 PR should include before/after benchmark subset results (how-to: website → AI CLI → Model Benchmark → "Running the Benchmark Yourself")

## Open questions for discussion

1. Stage-2 boundary: where should CLI operations capability stop relative to the Web UI?
2. Is the confidence-tiered auto-repair model (2.5) acceptable to operators, and what defaults are safe?
3. Skill Pack format: YAML-based declarative packs vs. structured markdown — preferences?
4. Which Stage-3 scenario has the most real-world demand from your usage?

I'll break Stage-1 items into sub-issues over the coming weeks and link them here. Feedback on prioritization is very welcome — especially from users running the CLI against real production configs.