51zero / 51zero/eel-sdk

CsvSource type conversion with custom schema

未关闭
#370 3 条评论 0 个 reaction 已指派 1 人 已被 @grewalr 认领 在 GitHub 查看
主要语言
Scala
星标
147
派生
32
PR 合并指标
30 天内没有已合并 PR

描述

From the project README - [CSV source part](https://github.com/51zero/eel-sdk#csv-source) I got the idea that type conversion for loaded CSV should be performed according to the specified schema.

But if I define a custom schema for a `CsvSource` which has columns with other types than `String` (`Int` for example), then the values in that column are still returned as `String`.

Is it intended behaviour, bug or it just haven't been implemented?

Runnable example:

```scala
import java.io.ByteArrayInputStream
import java.nio.charset.StandardCharsets
import io.eels.component.csv.CsvSource
import io.eels.schema._

object CsvSourceTypeConversionTest extends App {

val exampleCsvString =
"""A,B,C,D
|1,2.2,3,foo
|4,5.5,6,bar
""".stripMargin

val stream = new ByteArrayInputStream(exampleCsvString.getBytes(StandardCharsets.UTF_8))
val schema = new StructType(Vector(
Field("A", IntType.Signed),
Field("B", DoubleType),
Field("C", IntType.Signed),
Field("D", StringType)
))
val ds = new CsvSource(stream _, Some(schema)).toDataStream()
val firstRow = ds.iterator.toIterable.head
val firstRowA = firstRow.get("A")
println(firstRowA) // prints 1 as expected
println(firstRowA.getClass.getTypeName) // prints java.lang.String
assert(firstRowA == 1) // this assertion will fail because firstRowA is not an Int
}
```

贡献指南

这个仓库没有索引到贡献指南

调研方向

Look at the CsvSource implementation in the eel-sdk codebase, likely under io.eels.component.csv. Examine how the schema is applied during row parsing. The test case provided shows that values are returned as Strings despite Int/Double schema. Start by running the provided example to confirm the issue, then trace the data flow from CSV parsing to row creation. Check if type conversion logic exists or needs to be added. Verify by writing a test in the project's test suite.

由索引模型根据 Issue 内容生成。

评估

技术栈
scala
领域
data-engineering
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。