Spark: Extend V2 strategy support for view-related commands
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
Follow-up from https://github.com/apache/iceberg/pull/14984#discussion_r3737384668.
### What happens
`ResolveViews` resolves any `UnresolvedTableOrView` pointing at an Iceberg V2 view into a `ResolvedV2View`:
```scala
case u @ UnresolvedTableOrView(CatalogAndIdentifier(catalog, ident), _, _, _) =>
ViewUtil.loadView(catalog, ident).map(view => ResolvedV2View(...)).getOrElse(u)
```
`RewriteViewCommands` then rewrites the view commands Iceberg owns, at parse time, into Iceberg-aware nodes: `CreateView`, `DropView`, `RenameTable`, `DescribeRelation`, `ShowCreateTable`, `ShowTableProperties`, `ShowViews`, and `UnresolvedView`. For those, `ExtendedDataSourceV2Strategy` has a matching case and planning succeeds.
Commands outside that set — `SHOW COLUMNS`, `ANALYZE TABLE`, and similar — still resolve to a `ResolvedV2View`, but the strategy has no case for a bare `ResolvedV2View` and falls through to `case _ => Nil` (`ExtendedDataSourceV2Strategy.scala:193`). The result is a planner failure rather than a clear unsupported-operation error.
### Scope
This is not specific to the Spark 4.2 work. The same `UnresolvedTableOrView` case exists in `spark/v4.0` and `spark/v4.1` (`ResolveViews.scala:59` in both), so any Spark module supporting Iceberg V2 views is affected. It reflects existing Spark-side V2 view limitations, and was deliberately left out of scope in #14984.
Temporary views are not affected: they are resolved before the extension rule runs, as established in the PR discussion.
### Proposed work
- Decide, per command, whether Iceberg should add V2 strategy support or surface a clear unsupported-operation error naming the command and the view.
- Apply the outcome consistently across the `spark/v4.x` modules rather than only the newest one.
- Add coverage in `TestViews`, which currently exercises neither `SHOW COLUMNS` nor `ANALYZE TABLE` against a V2 view.
cc @manuzhang
Contributor guide
Research direction
Start with ResolveViews.scala in the spark/v4.0 and spark/v4.1 modules, then inspect ExtendedDataSourceV2Strategy.scala and the existing TestViews coverage. Determine the expected handling for SHOW COLUMNS, ANALYZE TABLE, and similar commands, apply it consistently across spark/v4.x, and add tests showing either successful planning or a clear unsupported-operation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100