[Subtask] feat(spark-connector): add GravitinoLakehouseRESTDiscoveryPlugin with the Lance provider
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 339
Description
### Describe the subtask
Add the format-agnostic discovery plugin in `spark-connector/spark-common`, and ship the Lance provider with it — Lance discovery needs no new server API, so it can land independently of the Iceberg listing endpoint.
**Plugin core**
- `GravitinoLakehouseRESTDiscoveryPlugin` fails fast if it is listed after `GravitinoSparkPlugin` in `spark.plugins`. Initializing first is what lets it treat every `spark.sql.catalog.*` entry it sees as user-written, with no markers and no state shared between the plugins.
- Snapshots `SparkConf` at init.
- Provider SPI keyed by format; a provider activates only when `spark.sql.gravitino.REST.uri` is set, so the URI doubles as the per-format switch and no `enable*` flag is needed. If a `uri` is set but its provider is not on the classpath, the plugin fails fast.
- `@DeveloperApi CatalogRegistrationPolicy` — `boolean shouldRegister(String format, String catalogName)` and `default String registeredCatalogName(String format, String catalogName)` — selected with `spark.sql.gravitino.REST.registrationPolicy`. The default implementation registers every advertised catalog under its advertised name.
- Precedence is enforced by the plugin, not by the policy: names already claimed by a user's `spark.sql.catalog.` are dropped before the policy runs; returned names are validated (duplicate, collides with user configuration, or not a valid Spark identifier -> fail fast); user per-catalog sub-key > plugin-generated key > global `catalogProperties.`.
- `GravitinoDriverPlugin` logs `Register catalog {} failed` with a stack trace when the name is already registered. With this plugin that fires once per discovered catalog and reads as an error; distinguish "already registered by user configuration" from a genuine failure.
**Lance provider**
- Prefix `spark.sql.gravitino.lanceREST.{uri, catalogProperties.}`.
- Lists catalogs with the Lance Namespace root list `GET {lance-rest-base}/v1/namespace/list` via `org.lance.namespace.client.apache.api.NamespaceApi` from the lance-spark bundle, following `page_token`. The server serves exactly the `lakehouse-generic` catalogs from Gravitino and rejects anything else, so the advertised names are by construction valid `parent` values.
- Generates, for each accepted catalog:
```text
spark.sql.catalog.=org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog..impl=rest
spark.sql.catalog..uri=
spark.sql.catalog..parent=
```
- Injects `org.lance.spark.extensions.LanceSparkSessionExtensions` into `spark.sql.extensions`, reusing the existing deduplication so a manually configured extension is not added twice.
- Generates no storage configuration: `lance.storage.*` static pass-through is unchanged.
- Unit tests (fake provider for the core; generated-configuration and precedence assertions), an integration test against the Gravitino Lance REST server, and a user-facing page under `docs/spark-connector/`.
Design doc: https://github.com/apache/gravitino/blob/main/design-docs/spark-rest-catalog-registration.md
### Parent issue
https://github.com/apache/gravitino/issues/11064
Contributor guide
Research direction
Start with the design doc and the existing GravitinoSparkPlugin and GravitinoDriverPlugin, then inspect spark-connector/spark-common. Use the fake-provider unit tests to verify provider activation, naming, precedence, and extension deduplication, and exercise the Lance provider against the Gravitino Lance REST server. Done means the plugin, Lance integration, tests, and a user-facing page under docs/spark-connector/ are complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, data-engineering, documentation, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100