apache / apache/iceberg

SparkSessionCatalog with JDBC catalog: SHOW TABLES IN ... returns error but table exists in JDBC catalog

Open
#10,003 14 comments 0 reactions 0 assignees View on GitHub
bug not-stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Apache Iceberg version

1.5.0 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

# The issue

Issue with SparkSessionCatalog when using it with a non-Hive catalog

## Repro

My expectations are based on the [documentation](https://iceberg.apache.org/docs/1.5.0/spark-configuration/#catalog-configuration):

```
org.apache.iceberg.spark.SparkSessionCatalog adds support for Iceberg tables to Spark's built-in catalog, and delegates to the built-in catalog for non-Iceberg tables
```

Note that `spark_catalog` is a `org.apache.iceberg.spark.SparkSessionCatalog`

### Test Set 1:

NOTE: `spark.sql.defaultCatalog = spark_catalog`

```
CREATE
OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS
SELECT
1 AS id,
TIMESTAMP('2000-01-01 01') AS ts,
'a' AS col1
```

```
show tables in test_schema
```

Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`

```
show tables in spark_catalog.test_schema
```

Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`

```
show tables in iceberg_catalog.test_schema
```

Expected: the table
Actual: ❌ `org.apache.iceberg.exceptions.NoSuchNamespaceException: Namespace does not exist: test_schema`

```
select * from test_schema.iceberg_table
```

Expected: the table content
Actual: ✅ the table content

```
select * from spark_catalog.test_schema.iceberg_table
```

Expected: the table content
Actual: ✅ the table content

```
select * from iceberg_catalog.test_schema.iceberg_table
```

Expected: the table content
Actual: ❌ `org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_NOT_FOUND]`

### Test Set 2:

I've tried setting the default catalog: `spark.sql.defaultCatalog = iceberg_catalog`

```
CREATE
OR replace TABLE test_schema.iceberg_table USING iceberg partitioned BY (DAY (ts), truncate (2, id)) AS
SELECT
1 AS id,
TIMESTAMP('2000-01-01 01') AS ts,
'a' AS col1
```

The result:

```
show tables in test_schema
```

Expected: the table
Actual: ✅ the table

```
show tables in spark_catalog.test_schema
```

Expected: the table
Actual: ❌ `org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: [SCHEMA_NOT_FOUND] The schema 'test_schema' cannot be found. Verify the spelling and correctness of the schema and catalog.`

```
show tables in iceberg_catalog.test_schema
```

Expected: the table
Actual: ✅ the table

```
select * from test_schema.iceberg_table
```

Expected: the table content
Actual: ✅ the table content

```
select * from spark_catalog.test_schema.iceberg_table
```

Expected: the table content
Actual: ❌ `org.apache.spark.sql.AnalysisException: [TABLE_OR_VIEW_NOT_FOUND] The table or view 'spark_catalog'.'test_schema'.'iceberg_table' cannot be found.`

```
select * from iceberg_catalog.test_schema.iceberg_table
```

Expected: the table content
Actual: ✅ the table content

# The Setup:

iceberg : 1.5.0
Java Version : 11.0.22 (Eclipse Adoptium)
Scala Version : version 2.12.18 (offical image with little tweaks)

| Config | value |
| --- | --- |
|spark.sql.catalog.iceberg_catalog | org.apache.iceberg.spark.SparkCatalog|
|spark.sql.catalog.iceberg_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO|
|spark.sql.catalog.iceberg_catalog.jdbc.password | *********(redacted)|
|spark.sql.catalog.iceberg_catalog.jdbc.schema-version | V1|
|spark.sql.catalog.iceberg_catalog.jdbc.useSSL | false|
|spark.sql.catalog.iceberg_catalog.jdbc.user | *********(redacted)|
|spark.sql.catalog.iceberg_catalog.jdbc.verifyServerCertificate | false|
|spark.sql.catalog.iceberg_catalog.table-default.write.metadata.delete-after-commit.enabled | true|
|spark.sql.catalog.iceberg_catalog.table-default.write.metadata.previous-versions-max | 50|
|spark.sql.catalog.iceberg_catalog.type | jdbc|
|spark.sql.catalog.iceberg_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog|
|spark.sql.catalog.iceberg_catalog.warehouse | gs://spark_warehouse_staging|
|spark.sql.catalog.spark_catalog | org.apache.iceberg.spark.SparkSessionCatalog|
|spark.sql.catalog.spark_catalog.io-impl | org.apache.iceberg.gcp.gcs.GCSFileIO|
|spark.sql.catalog.spark_catalog.jdbc.password | *********(redacted)|
|spark.sql.catalog.spark_catalog.jdbc.schema-version | V1|
|spark.sql.catalog.spark_catalog.jdbc.useSSL | false|
|spark.sql.catalog.spark_catalog.jdbc.user | *********(redacted)|
|spark.sql.catalog.spark_catalog.jdbc.verifyServerCertificate | false|
|spark.sql.catalog.spark_catalog.type | jdbc|
|spark.sql.catalog.spark_catalog.uri | jdbc:postgresql://localhost:5432/iceberg-catalog|
|spark.sql.catalog.spark_catalog.warehouse | gs://spark_warehouse_staging|
|spark.sql.catalogImplementation | hive|
|spark.sql.defaultCatalog | spark_catalog|
|spark.sql.extensions | org.apache.sedona.sql.SedonaSqlExtensions,org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions|

Contributor guide

Open the contributing guide

Research direction

Start by reading org.apache.iceberg.spark.SparkSessionCatalog and SparkCatalog, then reproduce the SHOW TABLES and SELECT queries using the listed JDBC and catalog settings. Done means the catalog consistently resolves the existing JDBC-backed namespace and table for the demonstrated qualified and unqualified queries, without the reported schema or table-not-found errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, java, postgresql, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.