Core: Expose Iceberg-managed view properties
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Problem
Spark 4.2 `SparkCatalog` must distinguish properties managed by Iceberg from properties supplied through Spark when replacing a view. Iceberg's `ViewBuilder.replace` merges properties, so Spark performs a best-effort follow-up removal for properties omitted from the replacement. Iceberg-managed properties must be excluded from those removals.
PR #14984 currently maintains this list in the Spark module:
```java
private static final Set ICEBERG_MANAGED_VIEW_PROPERTIES =
ImmutableSet.of(
ViewProperties.VERSION_HISTORY_SIZE,
ViewProperties.METADATA_COMPRESSION,
ViewProperties.WRITE_METADATA_LOCATION,
ViewProperties.REPLACE_DROP_DIALECT_ALLOWED);
```
The list is accurate today, but it duplicates knowledge from `org.apache.iceberg.view.ViewProperties`. If core adds another Iceberg-managed property and the Spark copy is not updated, a subsequent `CREATE OR REPLACE VIEW` can silently remove that setting.
`ViewProperties.COMMENT` is intentionally not preserved by this mechanism because Spark owns comment replacement.
### Proposed fix
- Define an authoritative immutable collection or classifier for Iceberg-managed view properties next to `ViewProperties`.
- Update Spark and other consumers to use the core definition rather than maintaining copies.
- Preserve the intentional treatment of `COMMENT` as an engine-owned replacement field.
- Add coverage that prevents newly introduced Iceberg-managed properties from being treated as removable Spark properties.
### Context
Identified while reviewing Spark 4.2 support in #14984: https://github.com/apache/iceberg/pull/14984#discussion_r3788259688
Contributor guide
Research direction
Start with org.apache.iceberg.view.ViewProperties and trace SparkCatalog's view-replacement path, including ViewBuilder.replace and the existing copy from PR #14984. Define the core-managed property source, update consumers to use it, preserve Spark ownership of COMMENT, and add coverage showing managed properties are not removed during replacement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100