apache / apache/arrow-rs

Implement `std::fmt::Write` for `StringViewBuilder` to permit non contiguous data to be written

Open
#6,373 5 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

DataFusion has an optimized version of `concat(col1, ...)` for `StringView` added by @devanbenz in https://github.com/apache/datafusion/pull/12224 that uses a `StringViewArrayBuilder` which is similar but not the same as `StringViewBuilder` in arrow: https://github.com/apache/datafusion/blob/9bc39a0522840ed90de2a4d23157de2e192cd00f/datafusion/functions/src/string/common.rs#L464-L536

The major differences are:
1. You can call `write` to incrementally build up each string and then call `append_offset` to create each string. `StringBuilder` requires each input to be a single contiguous string to call
2. You can avoid creating the null buffer and pass it in to the finish

**Describe the solution you'd like**
I would like the APIs in arrow-rs to be sufficiently complete that we could use the arrow-rs versions rather than our own custom versions in DataFusion

**Describe alternatives you've considered**
`StringBuilder` allows this like

```rust
use std::fmt::Write;
use arrow_array::builder::GenericStringBuilder;
let mut builder = GenericStringBuilder::::new();

// Write data in multiple `write!` calls
write!(builder, "foo").unwrap();
write!(builder, "bar").unwrap();
// The next call to append_value finishes the current string
// including all previously written strings.
builder.append_value("baz");

let array = builder.finish();
assert_eq!(array.value(0), "foobarbaz");
```

I think it would be cool to try and follow the same API for `StringViewBuilder` -- though note that API may be more complicated to ensure we don't get any performance regressions

**Additional context**
1. See https://github.com/apache/arrow-rs/pull/6372 for better docs of what StringBuilder does
2. https://github.com/apache/arrow-rs/issues/6347 lists more background

Contributor guide

Open the contributing guide

Research direction

Start by locating StringViewBuilder in arrow-rs and compare its intended behavior with GenericStringBuilder's std::fmt::Write example. Read datafusion/functions/src/string/common.rs around the StringViewArrayBuilder reference, then determine how append_value, append_offset, and finish should interact. Done means non-contiguous write! calls produce the expected StringView values without performance regressions, with focused coverage for the new API.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.