KEP-107: SparkClient SDK support for external storage and compute frameworks

### What you would like to be added?

This issue is meant to track the integration of the `SparkClient` SDK with external storage and compute layers.

Storage layers to explore include table formats such as Iceberg, Delta Lake and Hudi and storage backends such as S3, MinIO and GCS.

Compute layers to explore include acceleration frameworks such as Apache Celeborn, Gluten + Velox, DataFusion Comet and RAPIDS GPU.

The vision is to make the `SparkClient` SDK's configuration surface capable of supporting these frameworks by exposing existing CRD fields, adding dependency management, and providing helpers, thereby enabling support for advanced data engineering use cases. Building bespoke connectors or adapters for each framework is beyond the scope of this issue.

Starter points for discussion:
* Expose `hadoopConf` in the SDK
* Add a `packages` / `jars` parameter or `Dependencies` option in the `connect` API
* Provide a helper for composing `spark.sql.extensions`
* Add configuration profile presets
* Document common data engineering configurations

### Why is this needed?

The SparkClient works well for simple workloads:

```python
spark = client.connect(
num_executors=5,
resources_per_executor={"cpu": "4", "memory": "8Gi"},
spark_conf={"spark.sql.adaptive.enabled": "true"},
)
```

But production data engineering requires configurations like this, which are currently awkward, undocumented, and partially unsupported:

```python
# Reading from an Iceberg catalog on S3
spark = client.connect(
spark_conf={
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions",
"spark.sql.catalog.lakehouse": "org.apache.iceberg.spark.SparkCatalog",
"spark.sql.catalog.lakehouse.type": "rest",
"spark.sql.catalog.lakehouse.uri": "http://iceberg-rest:8181",
"spark.sql.catalog.lakehouse.warehouse": "s3a://warehouse/iceberg",
"spark.jars.packages": "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.9.1",
# S3 storage — hadoopConf is not exposed, so users must manually
# prefix every Hadoop property with spark.hadoop. in spark_conf
"spark.hadoop.fs.s3a.endpoint": "https://s3.amazonaws.com",
"spark.hadoop.fs.s3a.access.key": "...",
"spark.hadoop.fs.s3a.secret.key": "...",
},
)
```

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Start with the SparkClient connect API and the existing CRD fields referenced in the issue. Compare the proposed hadoopConf, dependency, spark.sql.extensions, and configuration-profile options, then determine a focused scope. Done should include an agreed configuration surface and documentation for common external storage and compute setups.

Written by the indexing model from the issue text.