apache / apache/datafusion

Spark-compatible CAST operation

Abierto
#11,201 11 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Rust
Estrellas
9.3k
Forks
2.4k
Merge medio
3 d 11 h
PR fusionados (30 d)
360

Descripción

### Is your feature request related to a problem or challenge?

We're looking to use DataFusion as a replacement for Spark for some workflows, through Spark -> Substrait -> DataFusion conversions. Lot of the functionality already works, and it's been nice to see DataFusion and Spark agree on most behavior we've tested so far. However, one place where we're seeing more differences is the CAST expression, for example on casting complex types into strings (, or casting strings into numbers (where Spark is more lenient).

One option I've considered is to use [Comet's cast](https://github.com/apache/datafusion-comet/blob/main/core/src/execution/datafusion/expressions/cast.rs), which I'd expect to be closer aligned with Spark (or at least on the way there). However, is there a way for me to replace/redirect the inbuilt Cast expression into using the Comet implementation?

Or would there be any other alternatives I could try?

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza leyendo Comet's core/src/execution/datafusion/expressions/cast.rs y compáralo con el comportamiento integrado de CAST de DataFusion. Determina si CAST puede redirigirse o reemplazarse, centrándote en las conversiones de tipos complejos a cadenas y en las conversiones permisivas de cadenas a números; se considerará terminado cuando exista un enfoque concreto compatible y un comportamiento compatible con Spark.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
rust, spark
Área
data-engineering
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.