Spark-compatible CAST operation
- Lenguaje dominante
- Rust
- Estrellas
- 9.3k
- Forks
- 2.4k
- Merge medio
- 3 d 11 h
- PR fusionados (30 d)
- 360
Descripción
### Is your feature request related to a problem or challenge?
We're looking to use DataFusion as a replacement for Spark for some workflows, through Spark -> Substrait -> DataFusion conversions. Lot of the functionality already works, and it's been nice to see DataFusion and Spark agree on most behavior we've tested so far. However, one place where we're seeing more differences is the CAST expression, for example on casting complex types into strings (, or casting strings into numbers (where Spark is more lenient).
One option I've considered is to use [Comet's cast](https://github.com/apache/datafusion-comet/blob/main/core/src/execution/datafusion/expressions/cast.rs), which I'd expect to be closer aligned with Spark (or at least on the way there). However, is there a way for me to replace/redirect the inbuilt Cast expression into using the Comet implementation?
Or would there be any other alternatives I could try?
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Guía de contribución
Línea de trabajo
Comienza leyendo Comet's core/src/execution/datafusion/expressions/cast.rs y compáralo con el comportamiento integrado de CAST de DataFusion. Determina si CAST puede redirigirse o reemplazarse, centrándote en las conversiones de tipos complejos a cadenas y en las conversiones permisivas de cadenas a números; se considerará terminado cuando exista un enfoque concreto compatible y un comportamiento compatible con Spark.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- rust, spark
- Área
- data-engineering
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 25/100