apache / apache/datafusion-comet

Utf8View and BinaryView (i.e., StringView in Arrow, colloquially German-style strings) support

Open
#1,403 1 comment 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

With the experimental native scans built on DataFusion's ParquetExec and our update to DataFusion 45, we have the opportunity to start adding support for StringView. I have started scoping out this work and would like to start aggregating findings here.

### Describe the potential solution

Project-level:
- Bump arrow-java version. We're currently on 16.0.0. I believe the view types were added in 17.0.0. I tested bumping to 18.2.0 and so far it doesn't seem too painful.

Java-side:
- Add support for decoding `Utf8View` and `BinaryView` to `CometVector`. I prototyped this [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L161) and [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L201) for Utf8View and BinaryView, respectively.

Native-side:
- Enable StringViewArray by default in query execution and Parquet reader. [We're using a recent enough DataFusion version that this is done already](https://github.com/apache/datafusion/pull/13101).
- planner.rs and serde.rs should generate Utf8View and BinaryView types when possible.
- Shuffle:
- Add support to hash_util.
- Add support to shuffle_writer (slot_size, etc.)

I'm sure there's more than this, and will continue adding as I find stuff broken in my proof-of-concept branch.

### Additional context

Related DataFusion blogs:

https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/
https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/

Contributor guide

Open the contributing guide

Research direction

Start with the CometVector and CometPlainVector prototype links to understand the Java decoding approach. Then inspect planner.rs, serde.rs, hash_util, and shuffle_writer, along with the Arrow version change, to map StringView handling across native scans, Parquet reading, and shuffle. Done means Utf8View and BinaryView work through all listed paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rust, scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.