apache / apache/parquet-testing
Add example Variant objects where field_id_size and field_offset_size differ
- Dominant language
- Python
- Stars
- 90
- Forks
- 79
- Avg merge
- 15d 20h
- Merged PRs (30d)
- 1
Description
The object `value_header` holds `field_offset_size_minus_one` in bits 0-1 and `field_id_size_minus_one` in bits 2-3. Writers compute the two independently -- field ID width from the largest field ID in the object, offset width from the size of the field data -- so the two routinely differ.
Every object example in `variant/` uses 1-byte field IDs and 1-byte field offsets. When the two are equal, the two header fields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly. The corpus therefore cannot detect that class of bug. arrow-dotnet shipped exactly this bug and validated cleanly against this corpus the whole time (apache/arrow-dotnet#420).
Two examples would close the gap:
- Few fields (1-byte field IDs) but more than 256 bytes of field data (2-byte field offsets) -- header byte `0x06`.
- A metadata dictionary of more than 256 entries (2-byte field IDs) containing a small object whose own field data is under 256 bytes (1-byte field offsets) -- header byte `0x12`.
Related to #78 but not covered by it: #78 is framed around field *count* and treats the two widths as one knob ("a 3 byte field id / offset"). The first
example above has only two fields. The second overlaps with #78's ">2^8 fields" case.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by inspecting the existing object examples in variant/ and how their headers and metadata dictionaries are represented. Add the two requested cases with differing field ID and offset widths, using header bytes 0x06 and 0x12; done means both examples are present and exercise independent decoding of the two widths.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100