Spark: Filter views from SparkCatalog.listTables
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Problem
`SparkCatalog.listTables` delegates directly to the underlying Iceberg catalog:
```java
return icebergCatalog.listTables(Namespace.of(namespace)).stream()
.map(ident -> Identifier.of(ident.namespace().levels(), ident.name()))
.toArray(Identifier[]::new);
```
Most Iceberg catalogs return only tables. However, `HiveCatalog` with `list-all-tables=true` returns every matching metastore entry, including views. As a result, `SparkCatalog.listTables` may expose views even though Spark's `TableCatalog.listTables` contract requires table identifiers only.
Spark 4.2 makes the distinction explicit through `RelationCatalog`: `listTableSummaries` returns tables only, while `listRelationSummaries` combines table and view summaries. `SparkCatalog.listTableSummaries` already subtracts identifiers returned by `listViews`, but `listTables` remains unfiltered.
This behavior predates Spark 4.2 and is not a regression in the Spark 4.2 support PR.
### Proposed fix
- Filter view identifiers from `SparkCatalog.listTables`, including the `HiveCatalog` `list-all-tables=true` configuration.
- Apply the correction consistently to supported Spark versions where the same implementation is present.
- Add coverage that creates both a table and a view in the same namespace and verifies that `listTables` returns only the table.
### Context
Recorded during review of Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259677
Contributor guide
Research direction
Locate SparkCatalog.listTables, listTableSummaries, and listViews, then inspect the existing Spark catalog test coverage. Add coverage that creates a table and view in one namespace and verifies listTables returns only the table across supported implementations or versions; run the relevant tests to confirm view identifiers are excluded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100