Spark-compatible CAST operation
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We're looking to use DataFusion as a replacement for Spark for some workflows, through Spark -> Substrait -> DataFusion conversions. Lot of the functionality already works, and it's been nice to see DataFusion and Spark agree on most behavior we've tested so far. However, one place where we're seeing more differences is the CAST expression, for example on casting complex types into strings (, or casting strings into numbers (where Spark is more lenient).
One option I've considered is to use [Comet's cast](https://github.com/apache/datafusion-comet/blob/main/core/src/execution/datafusion/expressions/cast.rs), which I'd expect to be closer aligned with Spark (or at least on the way there). However, is there a way for me to replace/redirect the inbuilt Cast expression into using the Comet implementation?
Or would there be any other alternatives I could try?
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading Comet's core/src/execution/datafusion/expressions/cast.rs and compare it with DataFusion's built-in CAST behavior. Define whether CAST can be redirected or replaced, focusing on complex-to-string and lenient string-to-number conversions; done requires a concrete supported approach and Spark-compatible behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100