apache / apache/gravitino

[Improvement] Define read and write DataType contracts

Open
#12,927 0 comments 0 reactions 1 assignee Claimed by @mchades View on GitHub
2.0.0 improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

`DataType` is currently reused by read responses and metadata write requests. Its shared deserializer preserves unknown strings and objects as `UnparsedType`. This is required for forward-compatible reads, but write APIs can silently persist unresolved type metadata.

Behavior also differs by surface:

- Function definitions are managed metadata.
- Table writes may pass through catalog-specific converters, while managed tables store types directly.
- Literal and default expressions also carry `DataType`.

This makes it unclear which inputs are structurally valid at the REST boundary and which are supported by a specific catalog.

### How should we improve?

Define three roles for the REST contract:

- Read-compatible `DataType`: native types, `ExternalType`, and `UnparsedType`.
- `WritableDataType`: recursively resolved native types or an explicit, non-blank `ExternalType`; never `UnparsedType`.
- Legacy selector inputs: retain read-compatible types when identifying existing metadata.

Keep wire-level resolvedness separate from catalog-specific capability validation. Accepting `WritableDataType` must not imply that every catalog supports the represented type. `ExternalType` should be the canonical form for intentionally supplied catalog- or engine-specific types; `UnparsedType` should remain a read-compatibility representation.

Roll this out by surface:

1. Function register and `AddDefinition`.
2. Table create and alter, including managed catalogs and connector compatibility.
3. Literal, default, and partition expressions and clients.

Existing metadata must remain readable and manageable, without automatic `UnparsedType` to `ExternalType` conversion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.