apache / apache/gravitino

[Subtask] Complete V3 type compatibility for the Spark connector

Open
#12,057 0 comments 0 reactions 1 assignee Claimed by @nevzheng View on GitHub
2.0.0 subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the subtask

Complete and document the five-family compatibility contract for Spark 3.3, 3.4, and 3.5.

**Compatibility outcomes**

- **Nanosecond timestamps:** Reject explicitly parameterized `timestamp(p)` and `timestamp_tz(p)` when `p > 6`, including precision 9. Preserve existing behavior for precision 6 and below. Reuse the work in #11941.
- **Variant:** Reject top-level and nested Variant fields because Spark 3.x has no lossless equivalent.
- **Unknown/Null:** Round-trip nullable Unknown only for Spark 3.4/3.5 Iceberg catalogs using Iceberg 1.11. Reject required fields, Spark 3.3, and non-Iceberg catalogs.
- **Geometry:** Reject top-level and nested fields rather than dropping WKB or CRS semantics.
- **Geography:** Reject top-level and nested fields rather than dropping WKB, CRS, or edge-algorithm semantics.

**Acceptance**

- Reproduce all five families through the connector's public create/conversion path and its alter/load paths where supported.
- A supported mapping must round-trip without losing type identity or parameters.
- An unsupported mapping must return `IllegalArgumentException` before external metadata mutation; no fallback to a narrower or opaque type is allowed.
- Cover nested types where the connector supports complex columns.
- Add focused converter/operation tests and an integration or no-side-effect test appropriate to the connector.
- Update the connector documentation with a five-family compatibility matrix and rejection rationale.
- Keep implementation commits atomic by connector and type family.

**Connector-specific verification**

Exercise the five families across supported Spark versions and catalog modes. Verify the supported Unknown mapping round-trips, every rejected case fails before table exposure or mutation, and precision 6-and-below timestamp behavior is unchanged. Run the shared Spark converter suite plus Spark 3.4 and 3.5 Iceberg converter suites.

**Documentation**

`docs/spark-connector/spark-catalog-iceberg.md`

### Parent issue

#12056

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.