DTStack / DTStack/chunjun

从mysql-x到clickhouse-x,mysql数据进行修改,ck数据库无变化

Open
#787 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

想咨询下我用的是sql脚本任务,从mysql-x到clickhouse-x时,当mysql数据发生删除或修改时,ck是没有变化的,请问这个有谁遇到过?有什么解决方法吗?

sql脚本如下:
CREATE TABLE source
(
id varchar,
name varchar,
code varchar,
ct bigint
) WITH (
'connector' = 'mysql-x',
'url' = 'jdbc:mysql://localhost:3306/test',
'table-name' = 'strtest3',
'username' = 'root',
'password' = '123'

,'scan.polling-interval' = '3000' --间隔轮训时间。非必填(不填为离线任务),无默认

,'scan.parallelism' = '2' -- 并行度大于1时,必须指定scan.partition.column。默认:1
,'scan.fetch-size' = '2' -- 每次从数据库中fetch大小。默认:1024条
,'scan.query-timeout' = '10' -- 数据库连接超时时间。默认:不超时

,'scan.partition.column' = 'ct' -- 多并行度读取的切分字段,多并行度下必需要设置。无默认 必须是主键而且必须自增
,'scan.partition.strategy' = 'range' -- 数据分片策略。默认:range,如果并行度大于1,且是增量任务或者间隔轮询,则会使用mod分片

,'scan.increment.column' = 'ct' -- 增量字段名称,必须是表中字段。非必填,无默认
,'scan.increment.column-type' = 'int' -- 增量字段类型。非必填,无默认
,'scan.start-location' = '20220510110122' -- 增量字段开始位置。非必填,无默认,如果没配置scan.increment.column,则不生效 从0行开始 跟着id走 如果上一次id为10,则输入10

,'scan.restore.columnname' = 'ct' -- 开启了cp,任务从sp/cp续跑字段名称。如果续跑,则会覆盖scan.start-location开始位置,从续跑点开始。非必填,无默认
,'scan.restore.columntype' = 'int' -- 开启了cp,任务从sp/cp续跑字段类型。非必填,无默认
);

CREATE TABLE sink
(
id varchar,
name varchar,
code varchar,
ct bigint
) WITH (
'connector' = 'clickhouse-x',
'url' = 'jdbc:clickhouse://xxxxxx:8123/test',
'table-name' = 'strtest',
-- 'username' = 'default',
-- 'password' = 'b6rCe7ZV',
'sink.buffer-flush.max-rows' = '1',
'sink.all-replace' = 'true'
);

insert into sink
select *
from source ;

ck脚本如下:
CREATE TABLE test.strtest (
`id` String,
`name` String,
`code` String,
`ct` UInt64
) ENGINE = MergeTree() ORDER BY ct SETTINGS index_granularity = 8192

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.