DataX最新版本v202309,新增HdfsReader支持parquet文件格式的问题
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
hive版本 : hive3
hdfs版本 : 3.1.5
场景 :
使用hive建表语句在建表时指定文件存储格式为parquet,压缩格式为snappy;
ds调度上新建任务,datax节点下配置文件中设置参数 "fileType": "PARQUET",运行任务提示以下错误,查阅datax的使用文档也并未找到新版本的使用说明以及报错处理
报错内容 :文件类型与用户配置的fileType类型不一致,请确认您配置的目录下面所有文件的类型均为[PARQUET]
有没有遇到相同问题的大佬,求助!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the HdfsReader parquet handling and the DataX usage documentation, then reproduce the reported error using Hive 3, HDFS 3.1.5, and a PARQUET fileType configuration. Trace why the detected file type differs from the configured value; done means the parquet scenario is correctly handled or the required configuration and error guidance are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100