apache / apache/datafusion-comet
Utf8View and BinaryView (i.e., StringView in Arrow, colloquially German-style strings) support
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
With the experimental native scans built on DataFusion's ParquetExec and our update to DataFusion 45, we have the opportunity to start adding support for StringView. I have started scoping out this work and would like to start aggregating findings here.
### Describe the potential solution
Project-level:
- Bump arrow-java version. We're currently on 16.0.0. I believe the view types were added in 17.0.0. I tested bumping to 18.2.0 and so far it doesn't seem too painful.
Java-side:
- Add support for decoding `Utf8View` and `BinaryView` to `CometVector`. I prototyped this [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L161) and [here](https://github.com/mbutrovich/datafusion-comet/blob/7c4eede25ba6672befa9beeb6f8c3e95dba7cc75/common/src/main/java/org/apache/comet/vector/CometPlainVector.java#L201) for Utf8View and BinaryView, respectively.
Native-side:
- Enable StringViewArray by default in query execution and Parquet reader. [We're using a recent enough DataFusion version that this is done already](https://github.com/apache/datafusion/pull/13101).
- planner.rs and serde.rs should generate Utf8View and BinaryView types when possible.
- Shuffle:
- Add support to hash_util.
- Add support to shuffle_writer (slot_size, etc.)
I'm sure there's more than this, and will continue adding as I find stuff broken in my proof-of-concept branch.
### Additional context
Related DataFusion blogs:
https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/
https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-2/
Contributor guide
Research direction
Start with the CometVector and CometPlainVector prototype links to understand the Java decoding approach. Then inspect planner.rs, serde.rs, hash_util, and shuffle_writer, along with the Arrow version change, to map StringView handling across native scans, Parquet reading, and shuffle. Done means Utf8View and BinaryView work through all listed paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rust, scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100