[Proposal] Spark 3.5+ baseline and Spark 4.x support

## Background

StreamPark's Spark integration (`-Pspark` profile) currently targets **Spark 3.1.2**, with version detection via `spark-submit --version` and shims loaded through `SparkShimsProxy`. Several gaps block StreamPark 3.0:

1. **Outdated baseline**: Spark 3.1.x is far behind current LTS lines; Spark 3.5+ is the practical minimum for new deployments.
2. **Spark 4.x**: Spark 4 requires **JDK 17+ on the cluster** (`spark-env.sh`), while StreamPark Console runs on JDK 11 — version detection and job JDK propagation must be decoupled.
3. **Remove Scala alignment**: Post–#4418, Spark shims jars no longer use `_2.12` artifact suffix; `SparkShimsProxy` must match Java-only jar names.
4. **Version detection fragility**: Relying solely on `spark-submit --version` fails when cluster JDK differs from the Console service JDK (common with Spark 4).

This issue tracks the **design and execution plan** for Spark version baseline changes in StreamPark 3.0.

**Implementation PR:** [#4420 [Spark] Baseline Spark 3.5+ and add Spark 4.x support](https://github.com/apache/streampark/pull/4420)
**Depends on:** [#4418 Remove Scala](https://github.com/apache/streampark/pull/4418) → [#4417 JDK 11 baseline](https://github.com/apache/streampark/pull/4417)
**Independent of:** [#4419 Flink version track](https://github.com/apache/streampark/pull/4419)
**Parent roadmap:** #4410

---

## Goals

- [ ] Baseline Spark at **3.5+**; reject Spark 2.x and 3.0–3.4 in `SparkVersion.checkVersion()`.
- [ ] Support **Spark 4.x** env registration and job submission.
- [ ] Resolve cluster-side `JAVA_HOME` from `spark-env.sh` (Spark 4 needs JDK 17+ on cluster).
- [ ] Update `SparkShimsProxy`, `YarnClient`, compile-time `spark.version`, and frontend hints.
- [ ] Document Console JDK vs Spark job JDK requirements.

## Non-Goals

- Supporting Spark 2.x / 3.0–3.4 in 3.0.
- Upgrading StreamPark Console to JDK 17 (Console stays JDK 11; Spark 4 jobs use cluster JDK 17).
- Migrating Spark modules to Java — handled by #4408 / #4418.
- Splitting into multiple PRs — **one consolidated PR** (#4420).

---

## Current Baseline (2.x)

| Area | Today |
|------|-------|
| `spark.version` (compile) | 3.1.2 |
| Supported runtime | Implicitly 2.x / 3.x (loose validation) |
| Spark 4.x | Not supported |
| Version detection | `spark-submit --version` stdout parsing |
| Job JDK | Not explicitly resolved from `spark-env.sh` |
| Shims jar matching | `_2.12` suffix expected |

---

## 3.0 Target State

| Area | Target |
|------|--------|
| **Supported 3.x** | 3.5, 3.5.x |
| **Supported 4.x** | 4.0, 4.x |
| **Rejected** | Spark 2.x, 3.0–3.4 |
| Compile-time `spark.version` | 3.5.4 |
| Version detection | `spark-core_*.jar` / `RELEASE` file first; fallback to `spark-submit --version` |
| Job JDK | `SparkEnvUtils` parses `JAVA_HOME` from `spark-env.sh`; propagated to `SparkLauncher` on YARN |
| Shims jar matching | No `_2.12` suffix (aligned with Remove Scala) |

---

## Proposed Architecture

### Two-tier JDK model

```
┌─────────────────────────────────────┐
│ StreamPark Console (JDK 11+) │ ← #4409 / #4417
└──────────────┬──────────────────────┘
│ submits / monitors
▼
┌─────────────────────────────────────┐
│ Spark Job Runtime │
│ JAVA_HOME from spark-env.sh │
│ Spark 3.5+ → JDK 11+ recommended │
│ Spark 4.x → JDK 17+ required │
└─────────────────────────────────────┘
```

### Key code changes (#4420)

| Component | Change |
|-----------|--------|
| `SparkVersion` | Accept 3.5+ and 4.x; parse `spark-core_*.jar` and `RELEASE` before `spark-submit` |
| `SparkEnvUtils` | New utility: read `JAVA_HOME` from `spark-env.sh` |
| `YarnClient` | Pass resolved cluster `JAVA_HOME` to `SparkLauncher` |
| `SparkShimsProxy` | Match shims jars without `_2.12` suffix |
| `pom.xml` | `spark.version` 3.1.2 → 3.5.4; add `commons-collections4.version=4.4` |
| Frontend `spark/home.ts` | Update env registration hints for 3.5+ / 4.x |

### Version detection priority

```
1. Parse spark-core_*.jar filename in $SPARK_HOME/jars/
2. Read $SPARK_HOME/RELEASE
3. Fallback: spark-submit --version (legacy path)
```

This avoids registration failures when Spark 4 is installed but `spark-submit` cannot run under the Console JVM.

---

## Execution Plan

Delivered as **one PR** (#4420) on branch `feature/spark-3.5-4-support`:

1. Update `SparkVersion`, add `SparkEnvUtils` with unit tests
2. Update `YarnClient`, `SparkShimsProxy`, root POM, frontend hints
3. CI green on JDK 11 + Java Spark modules (#4418 merged first)

**Can run in parallel with:** [#4419 Flink version track](https://github.com/apache/streampark/pull/4419) after #4418 merges.

---

## Risks & Mitigations

| Risk | Impact | Mitigation |
|------|--------|------------|
| Users on Spark 3.1–3.4 | High | 3.0 release notes; upgrade guide to 3.5+ |
| Spark 4 cluster JDK 17 not configured | Medium | `SparkEnvUtils` validation + clear Console error messages |
| YARN submission JDK mismatch | Medium | Propagate `JAVA_HOME` from `spark-env.sh` to `SparkLauncher` |
| Spark 4 API changes vs 3.5 compile dep | Medium | Shims proxy isolates version differences; integration test on 4.x |

---

## Acceptance Criteria

1. `SparkVersionTest` and `SparkEnvUtilsTest` pass on JDK 11.
2. `./mvnw clean compile -Pfast -pl streampark-spark -am` succeeds.
3. Console registers Spark 3.5 and 4.x homes; version detection correct without running `spark-submit` under Console JVM.
4. Spark job submits on YARN with Spark 4.x when `spark-env.sh` sets JDK 17.
5. Supported version matrix documented in upgrade guide.

---

## Open Questions

1. Spark **3.5.x** exact compile pin — 3.5.0 vs latest 3.5 LTS (currently **3.5.4** in #4420)?
2. K8s Spark submission: propagate `JAVA_HOME` via pod spec — in scope for #4420 or follow-up?
3. Spark module optional profile: document that `-Pspark` remains optional in 3.0?

---

## References

- Parent roadmap: #4410
- Remove Scala: #4408 / [#4418](https://github.com/apache/streampark/pull/4418)
- JDK 11 baseline: #4409 / [#4417](https://github.com/apache/streampark/pull/4417)
- Flink version track: #4421 / [#4419](https://github.com/apache/streampark/pull/4419) (parallel, independent)
- Implementation PR: [#4420](https://github.com/apache/streampark/pull/4420)

Start with implementation PR #4420 and inspect SparkVersion, SparkEnvUtils, YarnClient, SparkShimsProxy, pom.xml, and frontend/spark/home.ts. Run SparkVersionTest and SparkEnvUtilsTest, then the listed Maven compile command; done means the acceptance criteria pass for Spark 3.5 and 4.x, including YARN submission and documented version support.

Written by the indexing model from the issue text.