apache / apache/parquet-testing
Add example Variant values with larger numbers of fields / array elements
- Dominant language
- Python
- Stars
- 90
- Forks
- 79
- Avg merge
- 15d 20h
- Merged PRs (30d)
- 1
Description
- https://github.com/apache/parquet-testing/pull/76 adds example variant binary values
[Variant](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) is a binary format similar to JSON but allows more types in its value fields.
There are basic tests for nested structures object and list that have a few elements such as:
```json
[ 1, 2, 3, 4]
```
And
```json
{
"field1": 1
}
```
The examples added so far have few enough fields to be encoded with an 8 byte offset. However, the Variant spec includes variable sized offsets for objects with a larger number of fields or elements
```json
[ 1, 2, 3, 4, ... 4000000 ]
```
And
```json
{
"field1": 1
...
"field400000": 123
}
```
Here are the type of objects that do not have examples:
- object with 300 fields (more than 2^8 = 256) fields (requires a more than 1 byte num_elements)
- object with 66,000 fields (more than 2^16 = 65,536 fields) requires a 3 byte field id / offset
- object with 17M fields (more than 2^25 = 16,777,216 fields) requires a 4 byte field id / offset
This ticket tracks adding example Variant vales for such values. I did not add it to the original PR as I worried the binary items would be too large
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.