Iceberg-specific JDBC catalog properties should not be passed to driver
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
1.9.2 (latest release)
### Query engine
None
### Please describe the bug 🐞
Iceberg uses a number of config options prefixed with [`jdbc.`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.2/core/src/main/java/org/apache/iceberg/jdbc/JdbcCatalog.java#L77) internally, of which [`jdbc.schema-version`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.2/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L40-L41) is effectively mandatory, as you get a warning without it and need to set it in order for JdbcCatalog to support views.
The comment on `SCHEMA_VERSION_PROPERTY`, "property to control if view support is added to the existing database", is a bit misleading, because it also has to be set for view support to be added to newly-created tables, since it defaults to running `V0_CREATE_CATALOG_SQL`.
The problem is that Iceberg passes all properties prefixed with `jdbc.` to the driver when creating a new connection, stripping the prefix, and some JDBC drivers fail when they encounter unknown properties.
In [`JdbcClientPool.java`](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.2/core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java#L87-L88):
```java
@Override
protected Connection newClient() {
try {
Properties dbProps = JdbcUtil.filterAndRemovePrefix(properties, JdbcCatalog.PROPERTY_PREFIX);
return DriverManager.getConnection(dbUrl, dbProps);
} catch (SQLException e) {
throw new UncheckedSQLException(e, "Failed to connect: %s", dbUrl);
}
}
```
Ideally, there would be separate prefixes to distinguish properties meant to configure Iceberg's JdbcCatalog from those meant to be passed through to the driver.
To (mostly) maintain backwards compatibility, though, a compromise could be to filter out those Iceberg-specific properties from the set passed to the driver.
I believe there are just [three](https://github.com/apache/iceberg/blob/apache-iceberg-1.9.2/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L38-L44):
- `jdbc.strict-mode`
- `jdbc.schema-version`
- `jdbc.init-catalog-tables`
I can't think of a reason these would need to be passed to the driver.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start with JdbcClientPool.newClient in core/src/main/java/org/apache/iceberg/jdbc/JdbcClientPool.java and the JDBC property definitions and filtering in JdbcUtil.java and JdbcCatalog.java. Trace how the three Iceberg-specific properties are separated from driver properties, then verify that Iceberg options are no longer passed to the JDBC driver while ordinary connection properties still are.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100