apache / apache/parquet-testing
Add example nested Variant values that are not JSON encodeable
- Dominant language
- Python
- Stars
- 90
- Forks
- 79
- Avg merge
- 15d 20h
- Merged PRs (30d)
- 1
Description
- https://github.com/apache/parquet-testing/pull/76 adds example variant binary values
[Variant](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) is a binary format similar to JSON but allows more types in its value fields.
However, at the time of writing, I could not figure out how to create nested structures other than via JSON, which limits the type of the fields. I tried Apache spark, but to no avail.
Thus, we have values such as the following (valid JSON):
```json
{
"field1" : 123.455
}
```
However, there are no example values for data like this (where the field's value is a signed 8 bit integer):
```json
{
"field1" : 123u8,
}
```
This ticket tracks adding example values for Variants for nested data that is not JSON encodeable (e.g. with timestamp and other types of fields)
Contributor guide
No contributing guide indexed for this repository
Research direction
Review pull request 76 and the linked VariantEncoding.md specification first, then inspect how parquet-testing stores its existing example Variant binary values. Add nested examples containing non-JSON-encodable types such as signed 8-bit integers and timestamps, and confirm the examples are valid Variant data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100