DTStack / DTStack/chunjun

多并行度执行任务,报错,无法开启多并行度执行同步任务

Open
#1,482 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.1k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.

### What happened

多并行度执行任务,报错,无法开启多并行度
从mysql同步到hdfs上,无法开启多并行度执行
#

### What you expected to happen

配置信息
CREATE TABLE source
(
id bigint ,
branch_id bigint ,
xid varchar(100) ,
context varchar(128) ,
rollback_info string ,
log_status int ,
log_created timestamp ,
log_modified timestamp ,
ext varchar(100)
) WITH (

'scan.parallelism' = '10' -- 并行度大于1时,必须指定scan.partition.column。默认:1
,'scan.fetch-size' = '2048' -- 每次从数据库中fetch大小。默认:1024条
,'scan.query-timeout' = '10' -- 数据库连接超时时间。默认:1秒
,'scan.partition.column' = 'id' -- 多并行度读取的切分字段,多并行度下必需要设置。无默认
,'scan.partition.strategy' = 'range' -- 数据分片策略。默认:range
报错

split column must in columns;

at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:72)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.dtstack.chunjun.sql.parser.SqlParser.parseSql(SqlParser.java:66)
at com.dtstack.chunjun.Main.exeSqlJob(Main.java:145)
... 13 more
Caused by: java.lang.IllegalArgumentException: split column must in columns;

### How to reproduce

开启多并行度。执行同步任务即可

### Anything else

_No response_

### Version

master

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.