Evaluate replacing pyspark-connect with pyspark-client in [spark] extra

### Summary

`pip install "kubeflow[spark]"` currently pulls **`pyspark-connect==4.0.1`**, which depends on full **`pyspark`** and installs Spark JARs in the client environment.

Apache also publishes **`pyspark-client`**: a pure-Python Spark Connect client for **`spark.remote`** URIs (`sc://...`) without JVM/JAR dependencies.

Investigate whether Kubeflow **`SparkClient`** can switch to `pyspark-client` so client-only installs (notebooks, workbench images) avoid unused JVM artifacts and related scanner noise.

### Problem

Client images that only talk to a remote Spark Connect server still get:

- Larger installs (JARs + transitive deps)
- Implied Java/Py4J stack from full PySpark
- CVE findings on bundled Java libs (e.g. Netty, Jackson) that may never run locally

### Current SDK behavior

- Spark extra: `pyspark-connect==4.0.1`, `kubeflow-spark-api>=2.4.0` (`pyproject.toml`)
- Server alignment: `DEFAULT_SPARK_VERSION = "4.0.1"`, image `apache/spark:4.0.1`
- Connection path: `SparkSession.builder.remote(url).getOrCreate()` in connect and Kubernetes create paths
- Auth token: `spark.connect.authenticate.token` when `token` is passed
- No `SparkContext`, `_jvm`, `.master()`, or `spark-submit` usage under `kubeflow/spark/`
- Batch **`SparkApplication`** APIs use the Kubernetes backend; they do not require PySpark at runtime, but `[spark]` must still support interactive + batch installs

### Proposed change

Replace `pyspark-connect==4.0.1` with **`pyspark-client==4.0.1`** only if supported workflows pass.

**Known limitation if migrated:** `pyspark-client` supports **`spark.remote`** connection URIs only—not `spark.master` / local classic execution ([install docs](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)).

### Investigation checklist

1. `uv tree` before/after; `find .venv -name '*.jar'`
2. Swap dependency; `uv lock`; confirm `from pyspark.sql import SparkSession` still works
3. `make verify` + `make test-python`
4. E2E: `uv run pytest test/e2e/spark/ -v` (see `test/e2e/spark/README.md`)
5. Both connect paths:
- `SparkClient().connect(base_url="sc://...")`
- `SparkClient().connect()` after Kubernetes creates `SparkConnect`
6. Smoke: `spark.range(5).count()`, `spark.sql("SELECT 1")`, token + `spark_conf`
7. Run `examples/spark/*`; grep repo for local/JVM-only APIs
8. Confirm batch examples/APIs still work with `[spark]` install
9. Document size/JAR delta and any API gaps

### Failure handling

Classify failures (missing remote-client feature, accidental JVM API use, test infra, Apache bug, version skew). Do not reintroduce JVM deps to green a test. If local/master is required, keep `pyspark-connect` and document why.

### Acceptance criteria

**Migrate if green:**

- `[spark]` uses `pyspark-client==4.0.1`; lockfile updated
- No Spark JARs from `pip install "kubeflow[spark]"`
- Direct and Kubernetes-managed `connect()` work
- Unit + Spark E2E tests pass against Spark **4.0.1** server
- Docs/release notes state remote-only client; local/master not supported

**Block if not:**

- Reproducible failure tied to supported API
- Document blocker + conditions to retry; no dependency swap

### Test plan

```bash
make install-dev
make verify
make test-python
python -c 'from pyspark.sql import SparkSession; print(SparkSession)'
find .venv -type f -name '*.jar'
# After ./hack/e2e-setup-cluster.sh
uv run pytest test/e2e/spark/ -v
```

```python
from kubeflow.spark import SparkClient

client = SparkClient()
spark = client.connect(base_url="sc://spark-connect:15002")
assert spark.range(5).count() == 5
assert spark.sql("SELECT 1 AS value").first().value == 1
spark.stop()
```

Repeat equivalent ops on the Kubernetes-managed `connect()` path.

### Related

- [PySpark installation (pyspark-connect vs pyspark-client)](https://spark.apache.org/docs/latest/api/python/getting_started/install.html)
- [Spark Connect overview](https://spark.apache.org/docs/latest/spark-connect-overview.html)
- [SPARK-51212 / pyspark package split](https://github.com/apache/spark/pull/49946)
- [Kubeflow SDK Spark integration (#107)](https://github.com/kubeflow/sdk/issues/107)
- [SparkClient roadmap (ROADMAP.md)](https://github.com/kubeflow/sdk/blob/main/ROADMAP.md#spark-sdk)
- Spark E2E: `test/e2e/spark/README.md`