JdbcCatalog: getColumns/getTables with null catalog causes cross-schema false positive, leading to "Unknown column 'iceberg_type'" error
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Describe the bug
When multiple schemas/databases on the same MySQL instance each have an `iceberg_tables` table with different schema versions (V0 vs V1), `JdbcCatalog` incorrectly detects the `iceberg_type` column from a **different schema**, causing `schemaVersion` to be wrongly set to V1 while the actual table in the current connection's schema is V0.
This results in `SQLSyntaxErrorException: Unknown column 'iceberg_type' in 'where clause'` when creating or loading tables.
### Root Cause
In `JdbcCatalog.java`, both `updateSchemaIfRequired()` and `atomicCreateTable()` call `DatabaseMetaData.getColumns()` / `getTables()` with `null` as the catalog parameter:
```java
// updateSchemaIfRequired() — line 234
dbMeta.getColumns(
null, // ← scans ALL schemas on the server
null,
JdbcUtil.CATALOG_TABLE_VIEW_NAME, // "iceberg_tables" — '_' is a LIKE wildcard
JdbcUtil.RECORD_TYPE); // "iceberg_type"
// atomicCreateTable() — line 171
dbMeta.getTables(
null, // ← scans ALL schemas on the server
null,
name,
null);
```
Per JDBC spec, `null` for the catalog parameter means "no limitation" — the driver scans **all** schemas/databases on the server. When another schema on the same MySQL instance has an `iceberg_tables` table **with** the `iceberg_type` column (V1 schema), `getColumns()` returns that column even though the current connection's schema has a V0 table (without `iceberg_type`). This causes `schemaVersion` to be incorrectly set to `V1`, and subsequent V1 SQL queries fail with `Unknown column 'iceberg_type'`.
Additionally, `iceberg_tables` contains underscores (`_`), which are single-character wildcards in SQL `LIKE` matching, potentially causing further false matches.
### To Reproduce
**Environment:** A single MySQL instance with two schemas:
- `metaservice_db` — has `iceberg_tables` **with** `iceberg_type` column (V1 schema)
- `iceberg_catalog_db` — has `iceberg_tables` **without** `iceberg_type` column (V0 schema)
**Flink SQL:**
```sql
create catalog custom_iceberg_jdbc with (
'type' = 'iceberg',
'catalog-impl' = 'org.apache.iceberg.jdbc.JdbcCatalog',
'catalog-name' = 'custom_iceberg_jdbc',
'uri' = 'jdbc:mysql://host:port/iceberg_catalog_db',
'jdbc.user' = 'root',
'jdbc.password' = '***',
'jdbc.schema-version' = 'V0',
'warehouse' = 'hdfs://...'
);
create table if not exists custom_iceberg_jdbc.my_db.my_table (id int, name string);
```
Even though `jdbc.schema-version=V0` is set, the error still occurs:
```
Caused by: java.sql.SQLSyntaxErrorException: Unknown column 'iceberg_type' in 'where clause'
at org.apache.iceberg.jdbc.JdbcUtil.lambda$tableOrView$2(JdbcUtil.java:614)
at org.apache.iceberg.jdbc.JdbcUtil.tableOrView(JdbcUtil.java:602)
at org.apache.iceberg.jdbc.JdbcUtil.loadTable(JdbcUtil.java:641)
at org.apache.iceberg.jdbc.JdbcTableOperations.doRefresh(JdbcTableOperations.java:74)
```
### Verification on MySQL
```sql
-- Simulating getColumns(null, null, 'iceberg_tables', 'iceberg_type') — scans ALL schemas
SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME
FROM information_schema.COLUMNS
WHERE TABLE_NAME LIKE 'iceberg\_tables' AND COLUMN_NAME LIKE 'iceberg_type';
-- Result: metaservice_db | iceberg_tables | iceberg_type ← false positive from another schema!
-- Simulating getColumns('iceberg_catalog_db', null, 'iceberg_tables', 'iceberg_type') — limited to current schema
SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'iceberg_catalog_db' AND TABLE_NAME LIKE 'iceberg\_tables' AND COLUMN_NAME LIKE 'iceberg_type';
-- Result: (empty) ← correct, no iceberg_type column in this schema
-- Current schema's iceberg_tables table structure:
DESCRIBE iceberg_catalog_db.iceberg_tables;
-- Only 5 columns: catalog_name, table_namespace, table_name, metadata_location, previous_metadata_location
-- No iceberg_type column (V0 schema)
```
### Expected behavior
`JdbcCatalog` should only check for the `iceberg_type` column within the schema/database specified in the JDBC connection URL, not across all schemas on the server. The `jdbc.schema-version=V0` setting should be respected, and `schemaVersion` should remain V0 when the current schema's `iceberg_tables` table lacks the `iceberg_type` column.
### Proposed Fix
Use `conn.getCatalog()` to limit the `getColumns` / `getTables` search scope to the current schema, and escape underscores in the table name pattern to prevent LIKE wildcard matching:
```java
// updateSchemaIfRequired()
String catalog = conn.getCatalog();
String escapedTableName = JdbcUtil.CATALOG_TABLE_VIEW_NAME.replace("_", "\\_");
ResultSet typeColumn = dbMeta.getColumns(catalog, null, escapedTableName, JdbcUtil.RECORD_TYPE);
// atomicCreateTable()
String catalog = conn.getCatalog();
String escapedName = name.replace("_", "\\_");
ResultSet result = dbMeta.getTables(catalog, null, escapedName, null);
```
### Why existing tests didn't catch this
All existing `JdbcCatalog` tests use **SQLite** (in-memory or file-based). SQLite does not have the concept of multiple schemas/databases on the same server in the same way MySQL does, so `getColumns(null, null, ...)` in SQLite only sees the current database. The bug only manifests with multi-schema databases like MySQL or PostgreSQL when multiple schemas coexist on the same instance.
### Impact
This bug affects any deployment where multiple schemas on the same database instance have `iceberg_tables` tables with different schema versions. This is common in:
- Shared database instances (e.g., a metaservice schema and user catalog schemas coexisting on the same MySQL instance)
- Multi-tenant environments where different tenants use different schemas on the same database server
The same class of bug has been reported in other Iceberg language implementations:
- iceberg-go: #1188
- iceberg-rust: #2068
### Environment
- Iceberg version: 1.10.x (also confirmed present on `main` branch as of July 2026)
- Database: MySQL 8.0
- Compute engine: Flink 1.20
Contributor guide
Research direction
Start in JdbcCatalog.java at updateSchemaIfRequired() and atomicCreateTable(), then inspect the existing JdbcCatalog tests that use SQLite. Check metadata lookup behavior for the current connection catalog and table-name patterns, and add coverage for separate schemas with different table versions. Done means lookups do not detect tables or columns from another schema and the V0 case no longer queries iceberg_type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- database
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 66/100