[Improvement] Better integrate Gravitino with Iceberg REST and Lance REST to simplify compute engine configuration

### What would you like to be improved?

Today, when Spark queries Iceberg and Lance tables through Gravitino Lance&Iceberg REST server, the client still needs complex per-catalog configuration. In practice, users need to configure Iceberg REST and Lance REST catalogs separately in Spark, and each catalog requires its own settings such as catalog type, URI, extensions, and sometimes storage-related properties.

This creates several problems:

1. The client configuration is too complex for federated query scenarios.
2. Spark clients need to maintain catalog-by-catalog configuration outside Gravitino.
3. When a new catalog is added in Gravitino, the Spark client configuration also needs to be updated and refreshed.
4. The client-side operational burden grows as the number of catalogs increases.
5. This weakens the value of Gravitino as a unified metadata and access layer.

For example, to query Iceberg and Lance together from Spark, the client may need configuration like this:

```text
spark.plugins=org.apache.gravitino.spark.connector.plugin.GravitinoSparkPlugin
spark.sql.extensions=org.lance.spark.extensions.LanceSparkSessionExtensions
spark.sql.gravitino.uri=http://127.0.0.1:8090
spark.sql.gravitino.metalake=test
spark.sql.gravitino.enableIcebergSupport=true

spark.sql.catalog.iceberg_rest=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.iceberg_rest.type=rest
spark.sql.catalog.iceberg_rest.uri=http://127.0.0.1:9001/iceberg/
spark.sql.catalog.iceberg_rest.header.X-Iceberg-Access-Delegation=vended-credentials

spark.sql.catalog.lance=org.lance.spark.LanceNamespaceSparkCatalog
spark.sql.catalog.lance.impl=rest
spark.sql.catalog.lance.uri=http://127.0.0.1:9101/lance
spark.sql.catalog.lance.parent=lance_catalog
spark.sql.catalog.lance.storage.endpoint=http://127.0.0.1:9000
spark.sql.catalog.lance.storage.aws_allow_http=true
spark.sql.catalog.lance.storage.access_key_id=minioadmin
spark.sql.catalog.lance.storage.secret_access_key=minioadmin
spark.sql.catalog.lance.storage.region=us-east-1
```

This is hard to maintain on the client side. If a new Iceberg REST catalog or Lance REST catalog is added in Gravitino, the Spark client may also need new catalog entries or updated configuration before it can use that catalog.

Ideally, users should only need to configure the Gravitino server address in Spark, and then be able to run federated query and analysis across Iceberg and Lance tables managed by Gravitino, without manually defining each underlying REST catalog on the client side.

Example of the expected experience:

```text
spark.sql.gravitino.uri=http://127.0.0.1:8090
spark.sql.gravitino.metalake=test
```

Then Spark can directly discover and use Iceberg REST and Lance REST catalogs managed by Gravitino for unified query and analysis.

### How should we improve?

Design doc (merged): [`design-docs/spark-rest-catalog-registration.md`](https://github.com/apache/gravitino/blob/main/design-docs/spark-rest-catalog-registration.md)

Instead of teaching `GravitinoSparkPlugin` to translate catalog properties, discovery becomes server-authoritative and lives in a separate, format-agnostic plugin:

1. The Gravitino Iceberg REST server exposes a catalog-listing endpoint, so Spark never guesses catalog names (they match Gravitino names under `dynamic-config-provider` but not under `static-config-provider`). Lance needs no new endpoint — the Lance Namespace root list already returns exactly the catalogs that server serves.
2. A new `GravitinoLakehouseRESTDiscoveryPlugin` lists catalogs from each configured REST server at session startup and writes the `spark.sql.catalog.*` entries. Each lakehouse format plugs in as a provider (Lance, Iceberg), activated by its own `uri`, and a single `CatalogRegistrationPolicy` decides which catalogs are registered and under what Spark name.
3. The discovery plugin must precede `GravitinoSparkPlugin` in `spark.plugins` (validated, fail fast), so user-written `spark.sql.catalog.*` always wins and the existing plugin's behavior is unchanged. Users who do not add the new plugin see no difference.

### Task list

- [ ] #12524 — iceberg-rest-server: catalog-listing endpoint
- [ ] #12525 — spark-connector: `GravitinoLakehouseRESTDiscoveryPlugin` with the Lance provider
- [ ] #11305 — spark-connector: the Iceberg provider

#11305 was rewritten for this design; its original `engine-access-mode` proposal and the draft implementation #11838 are superseded.

Start with design-docs/spark-rest-catalog-registration.md, then read tasks #12524, #12525, and #11305 to understand the server endpoint and Spark connector providers. Check the existing GravitinoSparkPlugin behavior and the superseded #11838 context before working. Done means the design's catalog discovery and registration tasks are implemented without changing behavior for users who do not add the new plugin.

Written by the indexing model from the issue text.