apache / apache/arrow-rs

Provide some ability to bail out of nested ArrayBuilders

Open
#9,038 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When building a nested array using something like a StructBuilder, if you're halfway through a row/record, but then find that you can't continue building the array (perhaps you're parsing a streaming format into arrow), there's no easy way to either mark the whole row as null or the rest of the fields in the row as null easily. You have to continue to traverse your entire schema and push some dummy/null data into your child builders (recursively), and then at the end either append null or finish your row. This is very unergonomic, as you need special handling for this case in your entire parser.

**Describe the solution you'd like**
Discussed in discord, but something like a truncate_row api. There are a few questions to answer, should it somehow null out all the rest of the fields in your row, how does it interact with nested data (what if you're partway through some nested data at the moment/your child builders are in a halfway state)?

Is there a clean way to just ignore the state of your child builders and just dump that row?

**Describe alternatives you've considered**
Perhaps a cleaner option is providing an option to turn off length validation, `finish` your builder, and then slice off the last partial row?

Contributor guide

Open the contributing guide

Research direction

Start by examining the nested ArrayBuilder and StructBuilder APIs, especially how child-builder state and length validation are handled. Clarify the intended truncate_row behavior for partially built rows and nested data, then define completion criteria around nulling or discarding the partial row and add coverage for the chosen semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.