apache / apache/iceberg

Spark: Filter views from SparkCatalog.listTables

Open
#17,718 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Problem

`SparkCatalog.listTables` delegates directly to the underlying Iceberg catalog:

```java
return icebergCatalog.listTables(Namespace.of(namespace)).stream()
.map(ident -> Identifier.of(ident.namespace().levels(), ident.name()))
.toArray(Identifier[]::new);
```

Most Iceberg catalogs return only tables. However, `HiveCatalog` with `list-all-tables=true` returns every matching metastore entry, including views. As a result, `SparkCatalog.listTables` may expose views even though Spark's `TableCatalog.listTables` contract requires table identifiers only.

Spark 4.2 makes the distinction explicit through `RelationCatalog`: `listTableSummaries` returns tables only, while `listRelationSummaries` combines table and view summaries. `SparkCatalog.listTableSummaries` already subtracts identifiers returned by `listViews`, but `listTables` remains unfiltered.

This behavior predates Spark 4.2 and is not a regression in the Spark 4.2 support PR.

### Proposed fix

- Filter view identifiers from `SparkCatalog.listTables`, including the `HiveCatalog` `list-all-tables=true` configuration.
- Apply the correction consistently to supported Spark versions where the same implementation is present.
- Add coverage that creates both a table and a view in the same namespace and verifies that `listTables` returns only the table.

### Context

Recorded during review of Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259677

Contributor guide

Open the contributing guide

Research direction

Locate SparkCatalog.listTables, listTableSummaries, and listViews, then inspect the existing Spark catalog test coverage. Add coverage that creates a table and view in one namespace and verifies listTables returns only the table across supported implementations or versions; run the relevant tests to confirm view identifiers are excluded.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.