apache / apache/parquet-testing

Add example Variant values with larger numbers of fields / array elements

Open
#78 1 comment 0 reactions 1 assignee Claimed by @nssalian View on GitHub
Dominant language
Python
Stars
90
Forks
79
Avg merge
15d 20h
Merged PRs (30d)
1

Description

- https://github.com/apache/parquet-testing/pull/76 adds example variant binary values

[Variant](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) is a binary format similar to JSON but allows more types in its value fields.

There are basic tests for nested structures object and list that have a few elements such as:
```json
[ 1, 2, 3, 4]
```

And
```json
{
"field1": 1
}
```

The examples added so far have few enough fields to be encoded with an 8 byte offset. However, the Variant spec includes variable sized offsets for objects with a larger number of fields or elements

```json
[ 1, 2, 3, 4, ... 4000000 ]
```

And

```json
{
"field1": 1
...
"field400000": 123
}
```

Here are the type of objects that do not have examples:
- object with 300 fields (more than 2^8 = 256) fields (requires a more than 1 byte num_elements)
- object with 66,000 fields (more than 2^16 = 65,536 fields) requires a 3 byte field id / offset
- object with 17M fields (more than 2^25 = 16,777,216 fields) requires a 4 byte field id / offset

This ticket tracks adding example Variant vales for such values. I did not add it to the original PR as I worried the binary items would be too large

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.