多并行度执行任务,报错,无法开启多并行度执行同步任务
- Dominant language
- Java
- Stars
- 4.1k
- Forks
- 1.7k
- PR merge metrics
- No merged PRs in 30d
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/DTStack/chunjun/issues) and found no similar issues.
### What happened
多并行度执行任务,报错,无法开启多并行度
从mysql同步到hdfs上,无法开启多并行度执行
#
### What you expected to happen
配置信息
CREATE TABLE source
(
id bigint ,
branch_id bigint ,
xid varchar(100) ,
context varchar(128) ,
rollback_info string ,
log_status int ,
log_created timestamp ,
log_modified timestamp ,
ext varchar(100)
) WITH (
'scan.parallelism' = '10' -- 并行度大于1时,必须指定scan.partition.column。默认:1
,'scan.fetch-size' = '2048' -- 每次从数据库中fetch大小。默认:1024条
,'scan.query-timeout' = '10' -- 数据库连接超时时间。默认:1秒
,'scan.partition.column' = 'id' -- 多并行度读取的切分字段,多并行度下必需要设置。无默认
,'scan.partition.strategy' = 'range' -- 数据分片策略。默认:range
报错
split column must in columns;
at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:72)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at com.dtstack.chunjun.sql.parser.SqlParser.parseSql(SqlParser.java:66)
at com.dtstack.chunjun.Main.exeSqlJob(Main.java:145)
... 13 more
Caused by: java.lang.IllegalArgumentException: split column must in columns;
### How to reproduce
开启多并行度。执行同步任务即可
### Anything else
_No response_
### Version
master
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Assessment
This issue has not been assessed yet.