51zero / 51zero/eel-sdk

CsvSource type conversion with custom schema

Abierto
#370 3 comentarios 0 reacciones 1 asignado Reclamado por @grewalr Ver en GitHub
Lenguaje dominante
Scala
Estrellas
147
Forks
32
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

From the project README - [CSV source part](https://github.com/51zero/eel-sdk#csv-source) I got the idea that type conversion for loaded CSV should be performed according to the specified schema.

But if I define a custom schema for a `CsvSource` which has columns with other types than `String` (`Int` for example), then the values in that column are still returned as `String`.

Is it intended behaviour, bug or it just haven't been implemented?

Runnable example:

```scala
import java.io.ByteArrayInputStream
import java.nio.charset.StandardCharsets
import io.eels.component.csv.CsvSource
import io.eels.schema._

object CsvSourceTypeConversionTest extends App {

val exampleCsvString =
"""A,B,C,D
|1,2.2,3,foo
|4,5.5,6,bar
""".stripMargin

val stream = new ByteArrayInputStream(exampleCsvString.getBytes(StandardCharsets.UTF_8))
val schema = new StructType(Vector(
Field("A", IntType.Signed),
Field("B", DoubleType),
Field("C", IntType.Signed),
Field("D", StringType)
))
val ds = new CsvSource(stream _, Some(schema)).toDataStream()
val firstRow = ds.iterator.toIterable.head
val firstRowA = firstRow.get("A")
println(firstRowA) // prints 1 as expected
println(firstRowA.getClass.getTypeName) // prints java.lang.String
assert(firstRowA == 1) // this assertion will fail because firstRowA is not an Int
}
```

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Look at the CsvSource implementation in the eel-sdk codebase, likely under io.eels.component.csv. Examine how the schema is applied during row parsing. The test case provided shows that values are returned as Strings despite Int/Double schema. Start by running the provided example to confirm the issue, then trace the data flow from CSV parsing to row creation. Check if type conversion logic exists or needs to be added. Verify by writing a test in the project's test suite.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
scala
Área
data-engineering
Tipo de issue
Error
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bien especificado
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.