apache / apache/arrow-nanoarrow

Nested decoders do not propagate some types of required parameters into dictionary sub-decoders

Open
#936 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
249
Forks
68
Avg merge
2d 21h
Merged PRs (30d)
11

Description

Found from a codex review of https://github.com/apache/arrow-nanoarrow/pull/935:

- `ArrowIpcSetDecompressor()` won't propagate into dictionary decoders. We might need to add the ability for decompressors to clone themselves for this to work (or allow a reference to the parent or parent decompressor)
- Dictionary sub-decoders don't inherit the metadata version read by the parent decoder. All one tests files that have ever been written with a dictionary encoded union pre 1.0 will be read incorrectly (but it's a good point worth handling).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing ArrowIpcSetDecompressor() into dictionary sub-decoders and locating where the parent decoder reads metadata version information. Determine how dictionary decoders should receive decompressor and metadata-version state, then verify that decompressed data and pre-1.0 dictionary-encoded union files are read correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.