[Subtask] feat(spark-connector): add the Iceberg provider to GravitinoLakehouseRESTDiscoveryPlugin
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 315
Description
> **Note:** this issue originally tracked a `spark.sql.gravitino.lakehouse-iceberg.engine-access-mode` option inside `GravitinoSparkPlugin`. That approach was superseded by the merged design [`design-docs/spark-rest-catalog-registration.md`](https://github.com/apache/gravitino/blob/main/design-docs/spark-rest-catalog-registration.md), and the issue has been rewritten to track the Iceberg provider of the new discovery plugin. The earlier draft implementation is #11838.
### Describe the subtask
Add the Iceberg provider on top of the discovery plugin core (#12525), consuming the catalog-listing endpoint (#12524).
- Prefix `spark.sql.gravitino.icebergREST.{uri, catalogProperties.}`, disjoint from the existing plugin's keys; setting `uri` activates the provider.
- Lists catalogs with Iceberg's own `RESTClient` (`org.apache.iceberg.rest.HTTPClient`, the one `RESTCatalog` uses), not the Gravitino client: the target is the Iceberg REST server, its runtime is already on the classpath, and `ErrorResponse` is parsed the same way table calls do.
- Listing-call authentication is static forms only in V1 (`token`, `header.*`). A raw `RESTClient` does not run the `RESTCatalog` auth lifecycle — `AuthManager` loading and `AuthSession` token exchange happen a layer above it — so OAuth2-only deployments must additionally supply a static token for listing, while the generated catalogs still use the full OAuth2 flow. Reproducing that lifecycle with proper resource cleanup is future work.
- Generates, for each accepted catalog:
```text
spark.sql.catalog.=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog..type=rest
spark.sql.catalog..uri=
spark.sql.catalog..warehouse=
```
`warehouse` is always the advertised name, so a policy rename changes the Spark-facing name only and routing stays server-authoritative; the plugin logs any non-identity mapping.
- Injects `IcebergSparkSessionExtensions` into `spark.sql.extensions`, reusing the existing deduplication.
- Generates no storage configuration. Credential vending (`data-access=vended-credentials`) is the zero-configuration path; non-vending deployments supply credentials through the storage SDK's default credential chain or per-catalog sub-keys such as `spark.sql.catalog..s3.access-key-id`.
- Unit tests, plus integration tests covering: discovery to query against the Gravitino Iceberg REST server under both `static-config-provider` and `dynamic-config-provider`, plugin-ordering failure, a user-configured catalog left untouched, and both plugins configured together. Extend the `docs/spark-connector/` page added in #12525.
### Parent issue
https://github.com/apache/gravitino/issues/11064
Contributor guide
Research direction
Start with the discovery plugin core in #12525, the catalog-listing endpoint in #12524, and design-docs/spark-rest-catalog-registration.md. Add the Iceberg provider using Iceberg's RESTClient, then run the unit and integration scenarios listed in the issue. Done means accepted catalogs, extensions, authentication modes, ordering failures, user catalogs, and combined plugins are covered, with docs/spark-connector/ updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100