[Subtask] support iceberg rest catalog in spark-connector
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the subtask
support iceberg rest catalog in spark-connector.
1. Now, the `lakehouse-iceberg` catalog is created without registering the `rest service uri`

3. The spark-connector cannot get the `rest service uri` from the loaded catalog properties
4. I think that we should support registering `rest service uri` when creating a lakehouse-iceberg catalog and i have a draft plan here:
- **Web UI**:
If `enable-rest-service` is checked on the ui, the iceberg `rest service uri` must be specified

We may not need to consider supporting `rest catalog-backend` in the early stage:
1. iceberg rest service needs to support several current catalog-backend, so if rest catalog is used, `rest service uri` is required, regardless of which catalog-backend is chosen.
2. rest catalog-backend may be the gravitino service. Currently, the gravitino service does not implement a lock similar to hms. So i think that the rest catalog-backend implementation can lower the priority.
3. And we can start by supporting the registration of rest service uri into catalog properties.
- **com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogPropertiesMetadata**
```
public static final String ENABLE_REST_SERVICE = "enable-rest-service";
public static final String REST_SERVICE_URI = "rest-service-uri";
```
- **com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergConfig**
```
public static final ConfigEntry ENABLE_ICEBERG_REST_SERVICE =
new ConfigBuilder(ENABLE_REST_SERVICE)
.doc("Weather to enable Iceberg rest service")
.version(ConfigConstants.VERSION_0_5_0)
.booleanConf()
.create();
public static final ConfigEntry ICEBERG_REST_SERVICE_URI =
new ConfigBuilder(REST_SERVICE_URI)
.doc("The uri of Iceberg rest service")
.version(ConfigConstants.VERSION_0_5_0)
.stringConf()
.create();
```
- **spark-connector**
get the rest service uri directly from the iceberg catalog properties without manual configuration
```
public TableCatalog createAndInitSparkCatalog(
String name, CaseInsensitiveStringMap options, Map properties) {
Preconditions.checkArgument(
properties != null, "Iceberg Catalog properties should not be null");
HashMap all = new HashMap<>(options);
String enableIcebergRestService = properties.get(GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_ENABLE_REST_SERVICE);
if (StringUtils.isNotBlank(enableIcebergRestService) && "true".equalsIgnoreCase(enableIcebergRestService)) {
String icebergRestServiceUri = properties.get(GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_REST_SERVICE_URI);
Preconditions.checkArgument(StringUtils.isNotBlank(icebergRestServiceUri), "Iceberg Rest Service URI should not be empty.");
all.put(GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_TYPE, GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_TYPE_REST);
all.put(GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_URI, icebergRestServiceUri);
} else {
String catalogBackend = properties.get(GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_BACKEND);
Preconditions.checkArgument(
StringUtils.isNotBlank(catalogBackend), "Iceberg Catalog backend should not be empty.");
switch (catalogBackend.toLowerCase(Locale.ENGLISH)) {
case GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_BACKEND_HIVE:
initHiveProperties(catalogBackend, properties, all);
break;
case GravitinoSparkConfig.LAKEHOUSE_ICEBERG_CATALOG_BACKEND_JDBC:
initJdbcProperties(catalogBackend, properties, all);
break;
default:
// SparkCatalog does not support Memory type catalog
throw new IllegalArgumentException(
"Unsupported Iceberg Catalog backend: " + catalogBackend);
}
}
TableCatalog icebergCatalog = new SparkCatalog();
icebergCatalog.initialize(name, new CaseInsensitiveStringMap(all));
return icebergCatalog;
}
```
### Parent issue
https://github.com/datastrato/gravitino/issues/1571
Contributor guide
Research direction
Read IcebergCatalogPropertiesMetadata and IcebergConfig first, then trace createAndInitSparkCatalog in the spark-connector. Check the Web UI behavior described in the issue. Done means the REST service URI is registered in the Iceberg catalog properties, required when REST service is enabled, and used by the connector without manual configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100