apache / apache/datafusion-comet
[Spark 4.0] Add string collation support
Open
enhancement
spark 4.0
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
A major new feature of Spark 4.0.0 is string collation support. Unfortunately, DataFusion does not support string collation (https://github.com/apache/datafusion/issues/9192); therefore, Comet does not support it either.
### Describe the potential solution
In the short term, we likely need to fall back to Spark for any queries involving string collation.
### Additional context
https://www.databricks.com/blog/introducing-collations-databricks
https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-collation
Contributor guide
Assessment
This issue has not been assessed yet.