[Roadmap] StreamPark 3.0

## Overview

This issue is the **StreamPark 3.0 Roadmap** — the single source of truth for all planned breaking changes and major features targeted at the **3.0 release**.

| # | Theme | Summary | Design issue | Implementation PR(s) | Status |
|---|-------|---------|--------------|----------------------|--------|
| 1 | **JDK 11** | Console minimum runtime JDK 11; compile baseline 11 | #4409 | [#4417](https://github.com/apache/streampark/pull/4417) | ✅ **Done** — merged `dev` **2026-07-15** |
| 2 | **Remove dev framework** | Drop Flink/Spark dev traits and connector wrapper modules | #4431 | [#4432](https://github.com/apache/streampark/pull/4432) | ✅ **Done** — merged `dev` **2026-07-15** |
| 3 | **Remove Scala** | Java-only codebase; remove Scala toolchain | #4408 | [#4458](https://github.com/apache/streampark/pull/4458)–[#4467](https://github.com/apache/streampark/pull/4467), [#4471](https://github.com/apache/streampark/pull/4471) | 🔄 **Flink path done** — last merge **2026-08-03**; Spark deferred (#4453) |
| 4 | **Flink 1.17+ / 2.x** | Drop Flink 1.12–1.16; baseline 1.17–1.20; add 2.0–2.3 | #4421 | [#4419](https://github.com/apache/streampark/pull/4419) | ✅ **Done** — merged `dev` **2026-08-04** |
| 5 | **Spark 3.5+ / 4.x** | Baseline Spark 3.5+; add Spark 4.x support | #4422 | [#4420](https://github.com/apache/streampark/pull/4420) | ⏳ Pending |
| 6 | **Frontend Naive UI** | New `streampark-console-webapp-v2` on Naive UI + Vite | #4414 | [#4415](https://github.com/apache/streampark/pull/4415) | 🔄 In progress (PR open) |
| 7 | **PyFlink** | First-class PyFlink job lifecycle (submit, config, packaging, UI) | (this roadmap) | — | ⏳ Pending |

> **Note:** Original monolithic Remove Scala PR [#4418](https://github.com/apache/streampark/pull/4418) was **closed**; delivery split into class-by-class PRs #4458–#4467 + cleanup #4471 per community workflow (#4408).

---

## Current Baseline (2.x)

| Area | Today |
|------|-------|
| Console JDK | 8 (compile & runtime) |
| Flink shims | 1.12, 1.13, 1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.20 |
| Spark | 3.1.2 (`-Pspark` profile) |
| Backend language | ~343 Scala files + Java Console |
| Dev framework | `FlinkStreaming` / `FlinkTable` traits + connector wrappers |
| Frontend | Vue 3 + **Vite** + **Ant Design Vue 3** + Pinia |
| PyFlink | Partial — `.py` upload validation, `PythonDriver`, basic `PythonOptions` in client; no dedicated UI flow |

---

## 3.0 Target State

| Area | 3.0 Target | Status |
|------|------------|--------|
| Console JDK | **11+** required | ✅ **Done** ([#4417](https://github.com/apache/streampark/pull/4417), 2026-07-15) |
| Dev framework | **Removed** — users adopt native Flink/Spark APIs + official connectors | ✅ **Done** ([#4432](https://github.com/apache/streampark/pull/4432), 2026-07-15) |
| Flink | **1.17, 1.18, 1.19, 1.20, 2.0, 2.1, 2.2, 2.3** | ✅ **Done** ([#4419](https://github.com/apache/streampark/pull/4419), 2026-08-04) |
| Spark | **3.5+** and **4.x** (`-Pspark` profile) | ⏳ Pending ([#4420](https://github.com/apache/streampark/pull/4420)) |
| Backend | **Java only**, zero `.scala` | 🔄 Flink modules ✅ (2026-08-03); **17** `.scala` remain — Spark (#4453) + `common-scala-bridge` |
| Frontend | Vue 3 + Vite + **Naive UI** + Pinia (`webapp-v2`) | 🔄 In progress ([#4415](https://github.com/apache/streampark/pull/4415)) |
| PyFlink | Full support: env registration, venv packaging, submit on YARN/K8s/standalone, Console UI | ⏳ Pending |

---

## Delivery Tracks

### Track 1 — JDK 11 ✅ Completed ([#4409](https://github.com/apache/streampark/issues/4409) → [#4417](https://github.com/apache/streampark/pull/4417))

**Status:** Merged to `dev` on **2026-07-15** ([merge commit `e777acd`](https://github.com/apache/streampark/commit/e777acd6652cdccf82ae6cbe1ecab01fcced96fe)).

**Goal:** Console runs and builds on JDK 11.

**Delivered:**
- Maven compile baseline JDK 11; CI/Docker aligned to JDK 11
- Console runtime JDK 11+ enforcement, `javax.annotation-api`, JVM `--add-opens`
- `ClassLoaderUtils` JDK 9+ compatibility
- Dependency license registration for JDK 11 artifacts
- Yarn E2E Docker images bundle Eclipse Temurin JDK 11

---

### Track 2 — Remove dev framework ✅ Completed ([#4431](https://github.com/apache/streampark/issues/4431) → [#4432](https://github.com/apache/streampark/pull/4432))

**Status:** Merged to `dev` on **2026-07-15** ([merge commit `16f5fdc`](https://github.com/apache/streampark/commit/16f5fdc0dad7498b92e40d4f02a1da65b30ab545)).

**Goal:** Remove the StreamPark [development framework](https://streampark.apache.org/docs/framework/) and all connector wrapper modules; keep Console platform runtime.

**Delivered:**
- Removed `streampark-flink-core`, entire `streampark-flink-connector` tree, Spark dev traits, `streampark-spark-connector`
- Refactored Flink/Spark `SqlClient` to shims-based execution path

---

### Track 3 — Remove Scala 🔄 Flink path done ([#4408](https://github.com/apache/streampark/issues/4408))

**Goal:** Zero Scala source and zero Scala build toolchain.

**Delivery model:** Class-by-class migration (community workflow); original [#4418](https://github.com/apache/streampark/pull/4418) closed in favour of sequential PRs.

**Merged to `dev` (Flink + common):**

| PR | Module | Merged (UTC) |
|----|--------|--------------|
| [#4458](https://github.com/apache/streampark/pull/4458) | `streampark-common` | 2026-07-27 |
| [#4461](https://github.com/apache/streampark/pull/4461) | `streampark-flink-shims` | 2026-07-28 |
| [#4462](https://github.com/apache/streampark/pull/4462) | `streampark-flink-sqlclient` | 2026-07-29 |
| [#4463](https://github.com/apache/streampark/pull/4463) | `streampark-flink-packer` | 2026-07-31 |
| [#4464](https://github.com/apache/streampark/pull/4464) | `streampark-flink-proxy` | 2026-07-31 |
| [#4465](https://github.com/apache/streampark/pull/4465) | `streampark-flink-kubernetes` | 2026-08-02 |
| [#4467](https://github.com/apache/streampark/pull/4467) | `streampark-flink-client` | 2026-08-03 |
| [#4471](https://github.com/apache/streampark/pull/4471) | Post-migration build cleanup | 2026-08-03 |

**Remaining:** `streampark-spark` migration ([#4453](https://github.com/apache/streampark/issues/4453)); remove `streampark-common-scala-bridge` after Spark lands.

**Depends on:** ~~#4417~~ ✅ · ~~#4432~~ ✅

---

### Track 4 — Flink 1.17+ / 2.x ✅ Completed ([#4421](https://github.com/apache/streampark/issues/4421) → [#4419](https://github.com/apache/streampark/pull/4419))

**Goal:** Remove legacy Flink versions; add Flink 2.x shims.

- **Drop:** `streampark-flink-shims_flink-1.12` through `1.16`
- **Keep:** 1.17, 1.18, 1.19, 1.20
- **Add:** `streampark-flink-shims-base-v2`, **2.0 / 2.1 / 2.2 / 2.3** (Java, no `_2.12` suffix for 2.x)

**Status:** Merged to `dev` on **2026-08-04** ([merge commit `e770d2e`](https://github.com/apache/streampark/commit/e770d2e8e17fdeacddff62bf306f3235e136dca7)). Closes #4421.

**Depends on:** Flink Scala migration (#4461–#4467, #4471) ✅ on `dev`.

---

### Track 5 — Spark 3.5+ / 4.x ⏳ Pending ([#4422](https://github.com/apache/streampark/issues/4422) → [#4420](https://github.com/apache/streampark/pull/4420))

**Goal:** Bump Spark baseline; add Spark 4.x support.

**Status:** PR [#4420](https://github.com/apache/streampark/pull/4420) open.

**Depends on:** Spark Scala migration (#4453). **Independent of** #4419 (Flink).

---

### Track 6 — Frontend Naive UI 🔄 In progress ([#4414](https://github.com/apache/streampark/issues/4414) → [#4415](https://github.com/apache/streampark/pull/4415))

**Goal:** Deliver a new Naive UI console as `streampark-console-webapp-v2`.

**Status:** PR [#4415](https://github.com/apache/streampark/pull/4415) open. Parallel with backend tracks.

---

### Track 7 — PyFlink ⏳ Pending

**Goal:** Production-ready PyFlink support in Console.

**Dependency:** PyFlink UI should land in **Track 6** (`webapp-v2`).

---

## Cross-Track Dependency Graph

```
#4417 JDK 11 baseline ✅ (2026-07-15)
↓
#4432 Remove dev framework ✅ (2026-07-15)
↓
#4458–#4467 Scala → Java (Flink + common) ✅ (2026-07-27 … 2026-08-03)
#4471 Flink build cleanup ✅ (2026-08-03)
├──→ #4419 Flink 1.17+ / 2.x ✅ (2026-08-04)
├──→ #4453 Spark Scala → Java ⏳ (deferred)
└──→ #4420 Spark 3.5+ / 4.x ⏳ (after #4453)

#4415 Frontend Naive UI 🔄 — parallel

Track 7 PyFlink ⏳ — backend early; UI after #4415
```

---

## Recommended Merge Order

1. ~~[#4417](https://github.com/apache/streampark/pull/4417) — JDK 11 baseline~~ ✅ **2026-07-15**
2. ~~[#4432](https://github.com/apache/streampark/pull/4432) — Remove dev framework~~ ✅ **2026-07-15**
3. ~~[#4458](https://github.com/apache/streampark/pull/4458)–[#4467](https://github.com/apache/streampark/pull/4467) + [#4471](https://github.com/apache/streampark/pull/4471) — Flink Scala → Java~~ ✅ **2026-07-27 … 2026-08-03**
4. ~~[#4419](https://github.com/apache/streampark/pull/4419) — Flink 1.17+ / 2.x~~ ✅ **2026-08-04**
5. [#4453](https://github.com/apache/streampark/issues/4453) — Spark Scala → Java ← **next**
6. [#4420](https://github.com/apache/streampark/pull/4420) — Spark 3.5+ / 4.x (after #4453)
7. [#4415](https://github.com/apache/streampark/pull/4415) — Frontend Naive UI (can review in parallel)
8. **Track 7** — PyFlink
9. Release notes + upgrade guide

---

## 3.0 Breaking Changes Summary

| Change | User impact |
|--------|-------------|
| JDK 11+ required for Console | Upgrade deployment JVM |
| StreamPark dev framework removed | Migrate `FlinkStreaming`/`FlinkTable`/Spark traits to native Flink/Spark APIs |
| Connector wrapper modules removed | Use official Flink/Spark connectors |
| Flink 1.12–1.16 no longer supported | Re-register env on 1.17+ |
| Scala development API removed (Flink path done) | Rewrite Flink-side apps with Java API; Spark migration pending |
| Spark 3.5+ only (4.x supported) | Upgrade Spark clusters |
| Ant Design Vue UI replaced by Naive UI (`webapp-v2`) | Visual change; feature parity expected |
| PyFlink | New capability (additive) |

---

## Related Issues

- #4409 — JDK 11 proposal
- #4431 — Remove dev framework proposal
- #4408 — Remove Scala proposal
- #4421 — Flink 1.17+ / 2.x proposal
- #4422 — Spark 3.5+ / 4.x proposal
- #4414 — Frontend Naive UI proposal

This is a roadmap rather than a self-contained task; start by reading the pending tracks and their linked issues, especially #4453, #4420, and #4415. The payload names no files or tests; completion would require an agreed 3.0 scope and delivery of the remaining Spark, frontend, and PyFlink work.

Written by the indexing model from the issue text.