DTStack / DTStack/chunjun

mysql binlog实时同步增量数据到clickhouse时,ClickhouseOutputFormat - Writing records failed

Open
#1,033 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
背景:
测试使用binlog-x插件读取数据源端Mysql数据表中数据,获取mysql实时增量数据,同步写入Sink目标端Clickhouse中。

执行脚本如下:
${CHUNJUN_HOME}/bin/start-chunjun -mode local -jobType sync -job /xxx_job.json -chunjunDistDir ${CHUNJUN_HOME}/

xxx_job.json内容具体如下:

{
"job" : {
"content" : [ {
"reader" : {
"parameter" : {
"jdbcUrl" : "jdbc:mysql://localhost:3306/adc?useSSL=false",
"schema" : "adc",
"username" : "root",
"password" : "root",
"host" : "localhost",
"port" : 3306,
"cat" : "insert,delete,update",
"table" : ["white_list"]
},
"name" : "binlogreader"
},
"writer" : {
"parameter" : {
"column" : [
{
"name" : "id",
"index" : 0,
"isPart" : false,
"type" : "int64",
"key" : "id"
},
{
"name" : "item",
"index" : 1,
"isPart" : false,
"type" : "string",
"key" : "item"
}
],
"preSql":["truncate table white_list"],
"writeMode" : "insert",
"encoding" : "utf-8",
"fullColumnName" : [ "id", "item"],
"fullColumnType" : [ "INT64", "STRING"],
"connection" : [ {
"jdbcUrl" : "jdbc:clickhouse://localhost:1234/adc",
"table" : [ "white_list" ]
} ],
"schema": "adc",
"table": [ "white_list" ],
"password" : "root",
"username" : "root"
},
"name" : "clickhousewriter"
}
}
],
"setting" : {
"restore" : {
"maxRowNumForCheckpoint" : 0,
"isRestore" : false,
"restoreColumnName" : "",
"restoreColumnIndex" : 0
},
"errorLimit" : {
"record" : 100
},
"speed" : {
"bytes" : 0,
"channel" : 1
}
}
}
}

其中数据源端mysql数据表属性字段只有两列, 分别为 id:bigint; item:varchar;

clickhouser建表语句如下:

CREATE TABLE test.`white_list ` (
`id` Int64,
`item` Nullable(String)
) ENGINE = MergeTree
ORDER BY
tuple() SETTINGS index_granularity = 8192

==============================================================

问题描述:
![image](https://user-images.githubusercontent.com/45385723/177757079-d581a83f-15f4-4afa-a6d9-5ccc9764fa6c.png)

![image](https://user-images.githubusercontent.com/45385723/177757393-b3bd47fe-cbe2-46ee-958e-34f01624f26b.png)

![image](https://user-images.githubusercontent.com/45385723/177757499-18cb093a-d8ad-4652-bf11-17a33b22c9d3.png)

ERROR c.d.chunjun.connector.clickhouse.sink.ClickhouseOutputFormat - Writing records failed. com.dtstack.chunjun.throwable.NoRestartException: The dirty consumer shutdown, due to the consumed count exceed the max-consumed [0]

疑惑:
clickhouse writer模块中的column该如何来表示呢?

![image](https://user-images.githubusercontent.com/45385723/177757842-c5481e17-2778-4e36-9a62-0379ca6513bf.png)

江湖救急~~ 江湖救急 麻烦大佬解答下

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.