apache / apache/parquet-java

Make convert-csv work with the input filename which starts with a period or an numeric

未关闭
#2,339 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: CLI Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

I ran parquet-cli's `convert-csv` with an input file which name starts with a numeric character without `--schema` option and got the following error:

```Java

$ java -cp 'target/*:target/dependency/*' org.apache.parquet.cli.Main convert-csv 0sample.csv -o sample.parquet
Unknown error
shaded.parquet.org.apache.avro.SchemaParseException: Illegal initial character: 0sample
at shaded.parquet.org.apache.avro.Schema.validateName(Schema.java:1498)
at shaded.parquet.org.apache.avro.Schema.access$200(Schema.java:86)
at shaded.parquet.org.apache.avro.Schema$Name.(Schema.java:645)
at shaded.parquet.org.apache.avro.Schema.createRecord(Schema.java:182)
at shaded.parquet.org.apache.avro.SchemaBuilder$RecordBuilder.fields(SchemaBuilder.java:1805)
at org.apache.parquet.cli.csv.AvroCSV.inferSchemaInternal(AvroCSV.java:158)
at org.apache.parquet.cli.csv.AvroCSV.inferNullableSchema(AvroCSV.java:78)
at org.apache.parquet.cli.commands.ConvertCSVCommand.run(ConvertCSVCommand.java:160)
at org.apache.parquet.cli.Main.run(Main.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.parquet.cli.Main.main(Main.java:177)
```

This is because that `convert-csv` uses the input file name as the name for the output schema, while Avro requires its schema name to match the regex pattern `[A-Za-z_][A-Za-z0-9_]*`.
So users have to change the input file name or use the `--schema` option explicitly, but it's not so obvious from the error message.
It'd be nice if the message were improved, or the schema name were automatically replaced with valid characters to avoid this problem.

**Reporter**: [Kengo Seki](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sekikn) / @sekikn
**Assignee**: [Kengo Seki](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sekikn) / @sekikn
#### PRs and other links:
- [GitHub Pull Request #652](https://github.com/apache/parquet-mr/pull/652)

**Note**: *This issue was originally created as [PARQUET-1598](https://issues.apache.org/jira/browse/PARQUET-1598). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 org.apache.parquet.cli.csv.AvroCSV.inferSchemaInternal 开始,以堆栈跟踪和 ConvertCSVCommand.java 作为切入点。查看 pull request #652,以确定预期的解决方案是更清晰的错误,还是自动处理 schema 名称,然后添加测试覆盖,展示 convert-csv 会按规范处理以句点或数字开头的文件名。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
cli, data
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。