apache / apache/parquet-testing

Add example Variant objects where field_id_size and field_offset_size differ

Open Beginner friendly
#124 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
90
Forks
79
Avg merge
15d 20h
Merged PRs (30d)
1

Description

The object `value_header` holds `field_offset_size_minus_one` in bits 0-1 and `field_id_size_minus_one` in bits 2-3. Writers compute the two independently -- field ID width from the largest field ID in the object, offset width from the size of the field data -- so the two routinely differ.

Every object example in `variant/` uses 1-byte field IDs and 1-byte field offsets. When the two are equal, the two header fields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly. The corpus therefore cannot detect that class of bug. arrow-dotnet shipped exactly this bug and validated cleanly against this corpus the whole time (apache/arrow-dotnet#420).

Two examples would close the gap:

- Few fields (1-byte field IDs) but more than 256 bytes of field data (2-byte field offsets) -- header byte `0x06`.
- A metadata dictionary of more than 256 entries (2-byte field IDs) containing a small object whose own field data is under 256 bytes (1-byte field offsets) -- header byte `0x12`.

Related to #78 but not covered by it: #78 is framed around field *count* and treats the two widths as one knob ("a 3 byte field id / offset"). The first
example above has only two fields. The second overlaps with #78's ">2^8 fields" case.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting the existing object examples in variant/ and how their headers and metadata dictionaries are represented. Add the two requested cases with differing field ID and offset widths, using header bytes 0x06 and 0x12; done means both examples are present and exercise independent decoding of the two widths.

Written by the indexing model from the issue text.

Assessment

Domain
testing
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.