apache / apache/iceberg

Core: Expose Iceberg-managed view properties

Open
#17,719 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Problem

Spark 4.2 `SparkCatalog` must distinguish properties managed by Iceberg from properties supplied through Spark when replacing a view. Iceberg's `ViewBuilder.replace` merges properties, so Spark performs a best-effort follow-up removal for properties omitted from the replacement. Iceberg-managed properties must be excluded from those removals.

PR #14984 currently maintains this list in the Spark module:

```java
private static final Set ICEBERG_MANAGED_VIEW_PROPERTIES =
ImmutableSet.of(
ViewProperties.VERSION_HISTORY_SIZE,
ViewProperties.METADATA_COMPRESSION,
ViewProperties.WRITE_METADATA_LOCATION,
ViewProperties.REPLACE_DROP_DIALECT_ALLOWED);
```

The list is accurate today, but it duplicates knowledge from `org.apache.iceberg.view.ViewProperties`. If core adds another Iceberg-managed property and the Spark copy is not updated, a subsequent `CREATE OR REPLACE VIEW` can silently remove that setting.

`ViewProperties.COMMENT` is intentionally not preserved by this mechanism because Spark owns comment replacement.

### Proposed fix

- Define an authoritative immutable collection or classifier for Iceberg-managed view properties next to `ViewProperties`.
- Update Spark and other consumers to use the core definition rather than maintaining copies.
- Preserve the intentional treatment of `COMMENT` as an engine-owned replacement field.
- Add coverage that prevents newly introduced Iceberg-managed properties from being treated as removable Spark properties.

### Context

Identified while reviewing Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259688

Contributor guide

Open the contributing guide

Research direction

Start with org.apache.iceberg.view.ViewProperties and trace SparkCatalog's view-replacement path, including ViewBuilder.replace and the existing copy from PR #14984. Define the core-managed property source, update consumers to use it, preserve Spark ownership of COMMENT, and add coverage showing managed properties are not removed during replacement.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.