[Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
The Primitive Type Mappings table in the Variant canonical extension is the normative list of Arrow types allowed for `typed_value`. The only place the table is referenced from is the `typed_value` storage rule, so its whole role is defining legal `typed_value` types:
https://github.com/apache/arrow/blob/f10c93c000775c5fb30ca230f540358611091589/docs/source/format/CanonicalExtensions.rst?plain=1#L453
The table appears to be derived from the Variant *encoding* types table (https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types), but its role corresponds to the shredding spec's Shredded Value Types table (https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types), which defines what a `typed_value` column may be. The two tables differ, so the Arrow table currently allows storage with no valid Parquet shredded representation:
- `Uint8 -> Int16`, `Uint16 -> Int32`, `Uint32 -> Int64`: the shredding spec only permits signed integer `typed_value` columns, so unsigned Arrow storage can never round trip (`Arrow Uint -> Parquet INT -> Arrow Int`). A `Uint8` column also cannot represent the full value domain of its claimed variant type (int16), so it can only act as a partial storage that forces fallback to `value`. See the discussion in apache/arrow-rs#10417 (issue: apache/arrow-rs#10416).
- `Null -> Null`: the shredding table has no null row; a variant null must be encoded in the `value` column, so a `Null` typed_value column has no Parquet representation either.
In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned `typed_value` columns to Parquet with `INT(8/16/32, false)` annotations, which the shredding spec does not allow.
Proposal: re-derive the table from the shredding spec's Shredded Value Types table. Each row maps one variant type to its Parquet shredded type and the Arrow storage type(s) able to represent its full value domain. The table then defines reconstruction (a `typed_value` column of the listed Arrow type holds values of exactly the listed variant type) and the write-side lowering (the Parquet column is the only valid Parquet representation).
| Variant type | Parquet type | Arrow `typed_value` type |
|-----------------|-------------------------------------------------|-----------------------------------|
| boolean | BOOLEAN | Boolean |
| int8 | INT32, INT(8, true) | Int8 |
| int16 | INT32, INT(16, true) | Int16 |
| int32 | INT32 | Int32 |
| int64 | INT64 | Int64 |
| float | FLOAT | Float32 |
| double | DOUBLE | Float64 |
| decimal4, 1 <= P <= 9, 0 <= S <= P | INT32, DECIMAL(P, S) | Decimal32(P, S) |
| decimal8, 10 <= P <= 18, 0 <= S <= P | INT64, DECIMAL(P, S) | Decimal64(P, S) |
| decimal16, 19 <= P <= 38, 0 <= S <= P | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S)| Decimal128(P, S) |
| date | INT32, DATE | Date32 |
| time | INT64, TIME(false, MICROS) | Time64(us) |
| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
| string | BYTE_ARRAY, STRING | Utf8 / LargeUtf8 / Utf8View |
| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | UUID extension type |
The decimal precision bands follow the decimal table in [VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types): precision alone selects the physical type (the bands are disjoint, so the narrowest type is required), and scale must satisfy `0 <= S <= P`. P and S are shared across a row, so the bands constrain both the Parquet DECIMAL annotation and the Arrow type. Arrow otherwise permits negative scale and low-precision wide decimals, and Parquet permits both low-precision wide columns and BYTE_ARRAY decimals above precision 38.
This also resolves three existing ambiguities: the current `Time64` row gains its required unit (variant time is micros only), the timestamp rows gain explicit units, and the decimal rows gain their precision bounds.
Related: #50620, apache/parquet-format#591
### Component(s)
Format
Contributor guide
Assessment
This issue has not been assessed yet.