apache / apache/parquet-testing

Add example nested Variant values that are not JSON encodeable

Open
#77 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
90
Forks
79
Avg merge
15d 20h
Merged PRs (30d)
1

Description

- https://github.com/apache/parquet-testing/pull/76 adds example variant binary values

[Variant](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) is a binary format similar to JSON but allows more types in its value fields.

However, at the time of writing, I could not figure out how to create nested structures other than via JSON, which limits the type of the fields. I tried Apache spark, but to no avail.

Thus, we have values such as the following (valid JSON):

```json
{
"field1" : 123.455
}
```

However, there are no example values for data like this (where the field's value is a signed 8 bit integer):

```json
{
"field1" : 123u8,
}
```

This ticket tracks adding example values for Variants for nested data that is not JSON encodeable (e.g. with timestamp and other types of fields)

Contributor guide

No contributing guide indexed for this repository

Research direction

Review pull request 76 and the linked VariantEncoding.md specification first, then inspect how parquet-testing stores its existing example Variant binary values. Add nested examples containing non-JSON-encodable types such as signed 8-bit integers and timestamps, and confirm the examples are valid Variant data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.