apache / apache/gravitino

[Subtask] Complete V3 type compatibility for the Lance catalog

Open
#12,064 0 comments 0 reactions 1 assignee Claimed by @nevzheng View on GitHub
2.0.0 subtask
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### Describe the subtask

Complete and document the five-family compatibility contract for Lance and Arrow.

**Compatibility outcomes**

- **Nanosecond timestamps:** Round-trip exactly through Arrow `Timestamp(NANOSECOND, null/UTC)`; preserve native non-UTC zones as explicit external types.
- **Variant:** Reject Gravitino Variant before create or overwrite; preserve native Arrow extension types losslessly as external types.
- **Unknown/Null:** Round-trip nullable Unknown through Arrow Null; reject non-null fields.
- **Geometry:** Round-trip exactly through GeoArrow 0.2 WKB plus planar and CRS metadata.
- **Geography:** Round-trip exactly through GeoArrow WKB plus CRS and all five edge algorithms.

**Acceptance**

- Reproduce all five families through the connector's public create/conversion path and its alter/load paths where supported.
- A supported mapping must round-trip without losing type identity or parameters.
- An unsupported mapping must return `IllegalArgumentException` before external metadata mutation; no fallback to a narrower or opaque type is allowed.
- Cover nested types where the connector supports complex columns.
- Add focused converter/operation tests and an integration or no-side-effect test appropriate to the connector.
- Update the connector documentation with a five-family compatibility matrix and rejection rationale.
- Keep implementation commits atomic by connector and type family.

**Connector-specific verification**

Use native Lance schema round trips and verify a rejected Variant overwrite leaves the original dataset and metadata intact. Run the three module suites and native integration coverage for all five families.

**Documentation**

`docs/lakehouse-generic-lance-table.md`

### Parent issue

#12056

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.