SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Apache Iceberg version
1.5.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
# The issue
Issue with SparkSessionCatalog when using it with a non-Hive catalog
## Repro
My expectations are based on the [documentation](https://iceberg.apache.org/docs/1.5.0/spark-configuration/#catalog-configuration):
```
org.apache.iceberg.spark.SparkSessionCatalog adds support for Iceberg tables to Spark's built-in catalog, and delegates to the built-in catalog for non-Iceberg tables
```
Note that `spark_catalog` is a `org.apache.iceberg.spark.SparkSessionCatalog`
### Test Set 1:
NOTE: `spark.sql.defaultCatalog = spark_catalog`
```
CREATE
OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS
SELECT
1 AS id,
TIMESTAMP('2000-01-01 01') AS ts,
'a' AS col1
```
```
show tables in test_schema
```
Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`
```
show tables in spark_catalog.test_schema
```
Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`
```
show tables in iceberg_catalog.test_schema
```
Expected: the table
Actual: ❌ `org.apache.iceberg.exceptions.NoSuchNamespaceException: Namespace does not exist: test_schema`
```
select * from test_schema.iceberg_table
```
Expected: the table content
Actual: ✅ the table content
```
select * from spark_catalog.test_schema.iceberg_table
```
Expected: the table content
Actual: ✅ the table content
```
select * from iceberg_catalog.test_schema.iceberg_table
```
Expected: the table content
Actual: ❌ `org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_NOT_FOUND]`
### Test Set 2:
I've tried setting the default catalog: `spark.sql.defaultCatalog = iceberg_catalog`
```
CREATE
OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS
SELECT
1 AS id,
TIMESTAMP('2000-01-01 01') AS ts,
'a' AS col1
```
The result:
```
show tables in test_schema
```
Expected: the table
Actual: ✅ the table
```
show tables in spark_catalog.test_schema
```
Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`
```
show tables in iceberg_catalog.test_schema
```
Expected: the table
Actual: ✅ the table
```
select * from test_schema.iceberg_table
```
Expected: the table content
Actual: ✅ the table content
```
select * from spark_catalog.test_schema.iceberg_table
```
Expected: the table content
Actual: ❌ `org.apache.spark.sql.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view 'spark_catalog'.'test_schema'.'iceberg_table' cannot be found.`
```
select * from iceberg_catalog.test_schema.iceberg_table
```
Expected: the table content
Actual: ✅ the table content
# The Setup:
iceberg : 1.5.0
Java Version : 11.0.22 (Eclipse Adoptium)
Scala Version : version 2.12.18 (offical image with little tweaks)
| Config | value |
| --- | --- |
|spark.sql.catalog.iceberg_catalog | org.apache.iceberg.spark.SparkCatalog|
|spark.sql.catalog.iceberg_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO|
|spark.sql.catalog.iceberg_catalog.jdbc.password | *********(redacted)|
|spark.sql.catalog.iceberg_catalog.jdbc.schema-version | V1|
|spark.sql.catalog.iceberg_catalog.jdbc.useSSL | false|
|spark.sql.catalog.iceberg_catalog.jdbc.user | *********(redacted)|
|spark.sql.catalog.iceberg_catalog.jdbc.verifyServerCertificate | false|
|spark.sql.catalog.iceberg_catalog.table-default.write.metadata.delete-after-commit.enabled | true|
|spark.sql.catalog.iceberg_catalog.table-default.write.metadata.previous-versions-max | 50|
|spark.sql.catalog.iceberg_catalog.type | jdbc|
|spark.sql.catalog.iceberg_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog|
|spark.sql.catalog.iceberg_catalog.warehouse | gs://spark_warehouse_staging|
|spark.sql.catalog.spark_catalog | org.apache.iceberg.spark.SparkSessionCatalog|
|spark.sql.catalog.spark_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO|
|spark.sql.catalog.spark_catalog.jdbc.password | *********(redacted)|
|spark.sql.catalog.spark_catalog.jdbc.schema-version | V1|
|spark.sql.catalog.spark_catalog.jdbc.useSSL | false|
|spark.sql.catalog.spark_catalog.jdbc.user | *********(redacted)|
|spark.sql.catalog.spark_catalog.jdbc.verifyServerCertificate | false|
|spark.sql.catalog.spark_catalog.type | jdbc|
|spark.sql.catalog.spark_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog|
|spark.sql.catalog.spark_catalog.warehouse | gs://spark_warehouse_staging|
|spark.sql.catalogImplementation | hive|
|spark.sql.defaultCatalog | spark_catalog|
|spark.sql.extensions | org.apache.sedona.sql.SedonaSqlExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions|
Contributor guide
Research direction
Start by reading org.apache.iceberg.spark.SparkSessionCatalog and SparkCatalog, then reproduce the SHOW TABLES and SELECT queries using the listed JDBC and catalog settings. Done means the catalog consistently resolves the existing JDBC-backed namespace and table for the demonstrated qualified and unqualified queries, without the reported schema or table-not-found errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, java, postgresql, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100