[Subtask] Complete V3 type compatibility for the Spark connector
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the subtask
Complete and document the five-family compatibility contract for Spark 3.3, 3.4, and 3.5.
**Compatibility outcomes**
- **Nanosecond timestamps:** Reject explicitly parameterized `timestamp(p)` and `timestamp_tz(p)` when `p > 6`, including precision 9. Preserve existing behavior for precision 6 and below. Reuse the work in #11941.
- **Variant:** Reject top-level and nested Variant fields because Spark 3.x has no lossless equivalent.
- **Unknown/Null:** Round-trip nullable Unknown only for Spark 3.4/3.5 Iceberg catalogs using Iceberg 1.11. Reject required fields, Spark 3.3, and non-Iceberg catalogs.
- **Geometry:** Reject top-level and nested fields rather than dropping WKB or CRS semantics.
- **Geography:** Reject top-level and nested fields rather than dropping WKB, CRS, or edge-algorithm semantics.
**Acceptance**
- Reproduce all five families through the connector's public create/conversion path and its alter/load paths where supported.
- A supported mapping must round-trip without losing type identity or parameters.
- An unsupported mapping must return `IllegalArgumentException` before external metadata mutation; no fallback to a narrower or opaque type is allowed.
- Cover nested types where the connector supports complex columns.
- Add focused converter/operation tests and an integration or no-side-effect test appropriate to the connector.
- Update the connector documentation with a five-family compatibility matrix and rejection rationale.
- Keep implementation commits atomic by connector and type family.
**Connector-specific verification**
Exercise the five families across supported Spark versions and catalog modes. Verify the supported Unknown mapping round-trips, every rejected case fails before table exposure or mutation, and precision 6-and-below timestamp behavior is unchanged. Run the shared Spark converter suite plus Spark 3.4 and 3.5 Iceberg converter suites.
**Documentation**
`docs/spark-connector/spark-catalog-iceberg.md`
### Parent issue
#12056
Contributor guide
Assessment
This issue has not been assessed yet.