apache / apache/datafusion

Use optimized StringBuilders / BulkNullStringArrayBuilder in more places

Open
#22,148 3 comments 0 reactions 1 assignee Claimed by @neilconway View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

- a follow on to https://github.com/apache/datafusion/pull/22029

The idea is that now that we have some very optimized string builder APIs that generalize to the three different string types, we can reuse them in multiple kernels

At the moment the code is all in in the datafusion-functions crate: https://github.com/apache/datafusion/blob/7708aa2dc61271423a5c334bd2e2025b5e275133/datafusion/functions/src/strings.rs

However, that means they can't be used in other crates. I suggest we could put the string code in https://github.com/apache/datafusion/blob/0dfcd97a37e083e48aefc5267539ac453cc07b44/datafusion/physical-expr-common

This is consistent with things like String/BinaryMap:
https://github.com/apache/datafusion/blob/0dfcd97a37e083e48aefc5267539ac453cc07b44/datafusion/physical-expr-common/src/binary_map.rs#L40-L39

This might make it easier to and and reuse across crates

As @neilconway says:

Other places where these APIs should be useful:

* `initcap`
* `lower`, `upper`: at least for the Unicode code path; for ASCII, we might not beat the hand-optimized code added in #21980
* `translate`
* `reverse` (might need a slightly different API)
* `to_char` (might need a small API extension)
* `lpad`, `rpad` (needs a closer look)

If we make the builders accessible outside the current crate, some of the Spark functions could use these APIs, as well as `||` for `Utf8View` values.

_Originally posted by @neilconway in https://github.com/apache/datafusion/issues/22029#issuecomment-4382974325_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.