alibaba / alibaba/DataX

从clickhouse同步到clickhouse中,Map()类型的字段无法识别

Open
#1,833 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

```shell
2023-07-07 13:44:24.088 [0-0-0-reader] ERROR StdoutPlugincollector - com.alibaba.datax.common.exception.DataXException: code: [DBUtilErrorcare-121],Description:不支持的教据库类型,请注意查看datax已经支持的数据库类型以及数据库版本.].一您的配置文件中的列配置信息有误。因为Datax 不支持数据库读取这种字段类型
,字段名:[r0],字段名称:[2002],字段Java类型:[java.util.Map].请尝试使用数据库函数将其转换datax支持的类型 或者不同步该字段 。
at com. alibaba.datax. common.exception. DataXException. asDataXException (DataXException. java: 30) ~ [datax-common-0.0.1-SNAPSHOT. jar:na]
at com.alibaba. datax.plugin. reader. rdbmsreader . SubCommonRdbmsReader $Task . transportoneRecord (SubCommonRdbmsReader.java:145) ~ [rdbmsreader-0.0.1-SNAPSHOT. jar:na]
at com.alibaba. datax.plugin.rdbms. reader. CommondbmsReader$Task. startRead (CommondbmsReader. java: 209) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.rdbmsreader . RdbmsReader$Task. startRead (RdbmsReader. java: 84) [rdbmsreader-0.0.1-SNAPSHOT. jar:na]
at com.alibaba.datax.core.taskgroup.runner. ReaderRunner. run (ReaderRunner. java:57) [datax-core-0.0.1-SNAPSHOT. jar:na]
at java. lang. Thread. run (Thread. java:748) [na: 1.8.0_241]
2023-07-07 13:44:24.139 [0-0-0-reader] ERROR StdoutPlugincollector
```

远程clickhouse数据库中建了一张表,其中有几十个字段类型为Map(String,Decimal(32,8)),使用rdbmsread(已将clickhousewriter的驱动复制到rdbmsread/lib/目录下)读取时提示不支持该类型的字段。
项目自己拉了源码,修改了clickhouse驱动版本为0.4.6,其他表同步没问题。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with plugin/reader/rdbmsreader/SubCommonRdbmsReader.java, especially the transportoneRecord path named in the stack trace, and inspect how the ClickHouse 0.4.6 driver reports Map(String,Decimal(32,8)). Reproduce the rdbmsreader read against a table containing that column; done means the Map field is handled without the unsupported-type error and synchronization completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.