apache / apache/arrow-go

[Parquet] Unsigned Variant `typed_value` materializes as an invalid unsigned Parquet shredded column

Open
#1,202 2 comments 0 reactions 1 assignee Claimed by @dgvj-work View on GitHub
Type: bug
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

### Describe the bug, including details regarding any error messages, version, and platform.

## Description

Arrow Go permits unsigned Arrow integer types as `typed_value` fields in the `arrow.parquet.variant` extension type. When such an array is written to Parquet, the unsigned Arrow type is materialized directly as an unsigned Parquet integer column.

For example, an Arrow `Uint32` Variant `typed_value` produces:

```text
optional int32 typed_value (Int(bitWidth=32, isSigned=false));
```

This is not a valid shredded Variant type according to the Parquet Variant shredding specification. Its supported integer representations are:

- `int8` → `INT32 / INT(8, true)`
- `int16` → `INT32 / INT(16, true)`
- `int32` → `INT32`
- `int64` → `INT64`

Unsigned Parquet integer logical types are not supported.

The current Arrow canonical Variant extension specification permits unsigned Arrow storage types through widening mappings:

- `Uint8` → Variant `Int16`
- `Uint16` → Variant `Int32`
- `Uint32` → Variant `Int64`

## Interoperability test

The parquet-testing interoperability suite includes case 127, which expects `INTEGER(32,false)` to be rejected:

```json
{
"case_number": 127,
"test": "testUnsignedInteger",
"parquet_file": "case-127.parquet",
"error_message": "Unsupported shredded value type: INTEGER(32,false)"
}
```

Arrow Go currently skips that case with:

```go
s.T().Skip("Skipping case 127: test says uint32 should error, we just upcast to int64")
```

The Parquet column is not upcast to `int64`. It is materialized as Arrow `Uint32`, retained as `array.Uint32`, and written back to Parquet as `INT32 / INT(32,false)`.

Variant reconstruction may encode the value as a signed Variant integer, using `Int64` where necessary, but that does not make the shredded Parquet schema valid.

## Expected behavior

At minimum, Arrow Go should cast unsigned Arrow `typed_value` columns to their signed, widened Variant representations when writing Parquet:

- `Uint8` → signed `Int16`
- `Uint16` → signed `Int32`
- `Uint32` → signed `Int64`

The resulting Parquet `typed_value` columns must use the corresponding signed shredded types rather than unsigned integer logical annotations.

The more complete resolution is to drop unsigned `typed_value` support from the Arrow Variant extension type altogether. If the Arrow specification change below is accepted, Arrow Go can reject unsigned Variant `typed_value` schemas directly, without requiring fine-grained conversions in the Parquet writer.

The interoperability test for case 127 should no longer be skipped. Its skip comment should also be removed or corrected because no Parquet schema upcast currently occurs.

## Related Arrow specification change

The current Arrow canonical extension specification permits these unsigned mappings, creating a representation that cannot map directly to a conforming Parquet shredded schema.

The following PR proposes aligning Arrow’s Variant primitive mappings with the Parquet shredding specification by removing unsupported mappings such as unsigned integers and Null:

- https://github.com/apache/arrow/pull/50810
- Related issue: https://github.com/apache/arrow/issues/50622

## Relevant links

- [Parquet Variant shredded-value types](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types)
- [Current Arrow canonical Variant primitive mappings](https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings)
- [Proposed Arrow specification correction](https://github.com/apache/arrow/pull/50810)
- [parquet-testing case 127](https://github.com/apache/parquet-testing/blob/e7d32dac5b4cbf017fefd598a03686f53370b292/shredded_variant/cases.json#L754-L757)
- [Arrow Go skips case 127](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/variant_test.go#L295-L304)
- [`INTEGER(32,false)` maps to Arrow `Uint32`](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/schema.go#L433-L454)
- [The shredded reader retains `array.Uint32`](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L646-L663)
- [Arrow Go’s Variant mappings include unsigned types](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L1195-L1206)
- [The shredded builder accepts unsigned Arrow builders](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/arrow/extensions/variant.go#L1456-L1486)
- [The writer emits unsigned Parquet annotations](https://github.com/apache/arrow-go/blob/986b027408e1d973cb78023fc74a2a1bc361c6a9/parquet/pqarrow/schema.go#L624-L645)

Generated with codex and proof read

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.