从clickhouse同步到clickhouse中,Map()类型的字段无法识别
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
```shell
2023-07-07 13:44:24.088 [0-0-0-reader] ERROR StdoutPlugincollector - com.alibaba.datax.common.exception.DataXException: code: [DBUtilErrorcare-121],Description:不支持的教据库类型,请注意查看datax已经支持的数据库类型以及数据库版本.].一您的配置文件中的列配置信息有误。因为Datax 不支持数据库读取这种字段类型
,字段名:[r0],字段名称:[2002],字段Java类型:[java.util.Map].请尝试使用数据库函数将其转换datax支持的类型 或者不同步该字段 。
at com. alibaba.datax. common.exception. DataXException. asDataXException (DataXException. java: 30) ~ [datax-common-0.0.1-SNAPSHOT. jar:na]
at com.alibaba. datax.plugin. reader. rdbmsreader . SubCommonRdbmsReader $Task . transportoneRecord (SubCommonRdbmsReader.java:145) ~ [rdbmsreader-0.0.1-SNAPSHOT. jar:na]
at com.alibaba. datax.plugin.rdbms. reader. CommondbmsReader$Task. startRead (CommondbmsReader. java: 209) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.rdbmsreader . RdbmsReader$Task. startRead (RdbmsReader. java: 84) [rdbmsreader-0.0.1-SNAPSHOT. jar:na]
at com.alibaba.datax.core.taskgroup.runner. ReaderRunner. run (ReaderRunner. java:57) [datax-core-0.0.1-SNAPSHOT. jar:na]
at java. lang. Thread. run (Thread. java:748) [na: 1.8.0_241]
2023-07-07 13:44:24.139 [0-0-0-reader] ERROR StdoutPlugincollector
```
远程clickhouse数据库中建了一张表,其中有几十个字段类型为Map(String,Decimal(32,8)),使用rdbmsread(已将clickhousewriter的驱动复制到rdbmsread/lib/目录下)读取时提示不支持该类型的字段。
项目自己拉了源码,修改了clickhouse驱动版本为0.4.6,其他表同步没问题。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with plugin/reader/rdbmsreader/SubCommonRdbmsReader.java, especially the transportoneRecord path named in the stack trace, and inspect how the ClickHouse 0.4.6 driver reports Map(String,Decimal(32,8)). Reproduce the rdbmsreader read against a table containing that column; done means the Map field is handled without the unsupported-type error and synchronization completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100