apache / apache/iceberg

Spark: Extend V2 strategy support for view-related commands

Open
#17,577 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

Follow-up from https://github.com/apache/iceberg/pull/14984#discussion_r3737384668.

### What happens

`ResolveViews` resolves any `UnresolvedTableOrView` pointing at an Iceberg V2 view into a `ResolvedV2View`:

```scala
case u @ UnresolvedTableOrView(CatalogAndIdentifier(catalog, ident), _, _, _) =>
ViewUtil.loadView(catalog, ident).map(view => ResolvedV2View(...)).getOrElse(u)
```

`RewriteViewCommands` then rewrites the view commands Iceberg owns, at parse time, into Iceberg-aware nodes: `CreateView`, `DropView`, `RenameTable`, `DescribeRelation`, `ShowCreateTable`, `ShowTableProperties`, `ShowViews`, and `UnresolvedView`. For those, `ExtendedDataSourceV2Strategy` has a matching case and planning succeeds.

Commands outside that set — `SHOW COLUMNS`, `ANALYZE TABLE`, and similar — still resolve to a `ResolvedV2View`, but the strategy has no case for a bare `ResolvedV2View` and falls through to `case _ => Nil` (`ExtendedDataSourceV2Strategy.scala:193`). The result is a planner failure rather than a clear unsupported-operation error.

### Scope

This is not specific to the Spark 4.2 work. The same `UnresolvedTableOrView` case exists in `spark/v4.0` and `spark/v4.1` (`ResolveViews.scala:59` in both), so any Spark module supporting Iceberg V2 views is affected. It reflects existing Spark-side V2 view limitations, and was deliberately left out of scope in #14984.

Temporary views are not affected: they are resolved before the extension rule runs, as established in the PR discussion.

### Proposed work

- Decide, per command, whether Iceberg should add V2 strategy support or surface a clear unsupported-operation error naming the command and the view.
- Apply the outcome consistently across the `spark/v4.x` modules rather than only the newest one.
- Add coverage in `TestViews`, which currently exercises neither `SHOW COLUMNS` nor `ANALYZE TABLE` against a V2 view.

cc @manuzhang

Contributor guide

Open the contributing guide

Research direction

Start with ResolveViews.scala in the spark/v4.0 and spark/v4.1 modules, then inspect ExtendedDataSourceV2Strategy.scala and the existing TestViews coverage. Determine the expected handling for SHOW COLUMNS, ANALYZE TABLE, and similar commands, apply it consistently across spark/v4.x, and add tests showing either successful planning or a clear unsupported-operation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.