Provide some ability to bail out of nested ArrayBuilders
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When building a nested array using something like a StructBuilder, if you're halfway through a row/record, but then find that you can't continue building the array (perhaps you're parsing a streaming format into arrow), there's no easy way to either mark the whole row as null or the rest of the fields in the row as null easily. You have to continue to traverse your entire schema and push some dummy/null data into your child builders (recursively), and then at the end either append null or finish your row. This is very unergonomic, as you need special handling for this case in your entire parser.
**Describe the solution you'd like**
Discussed in discord, but something like a truncate_row api. There are a few questions to answer, should it somehow null out all the rest of the fields in your row, how does it interact with nested data (what if you're partway through some nested data at the moment/your child builders are in a halfway state)?
Is there a clean way to just ignore the state of your child builders and just dump that row?
**Describe alternatives you've considered**
Perhaps a cleaner option is providing an option to turn off length validation, `finish` your builder, and then slice off the last partial row?
Contributor guide
Research direction
Start by examining the nested ArrayBuilder and StructBuilder APIs, especially how child-builder state and length validation are handled. Clarify the intended truncate_row behavior for partially built rows and nested data, then define completion criteria around nulling or discarding the partial row and add coverage for the chosen semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100