alibaba / alibaba/DataX

sqlserver to databend报错

Open
#1,791 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

[root@mysql-80-24 DataX]# python ./target/datax/datax/bin/datax.py ./sqlserver_xx.json
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-05-21 07:38:37.762 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-05-21 07:38:37.765 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-05-21 07:38:37.798 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-05-21 07:38:37.804 [main] INFO Engine - the machine info =>

osInfo: Red Hat, Inc. 1.8 25.372-b07
jvmInfo: Linux amd64 3.10.0-1160.24.1.el7.x86_64
cpu num: 16

totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1

GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB

2023-05-21 07:38:37.818 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"sqlserverreader",
"parameter":{
"column":[
"a"
],
"connection":[
{
"jdbcUrl":[
"jdbc:sqlserver://xxx:xx;DatabaseName=test"
],
"table":[
"s"
]
}
],
"password":"******",
"username":"test_user"
}
},
"writer":{
"name":"databendwriter",
"parameter":{
"column":[
"a"
],
"connection":[
{
"jdbcUrl":"jdbc:databend://xx:8000/migrate_db",
"table":"s"
}
],
"maxBatchRows":65536,
"maxBatchSize":134217728,
"password":"*********",
"postSql":[

],
"preSql":[

],
"username":"xx"
}
}
}
],
"setting":{
"speed":{
"channel":""
}
}
}

2023-05-21 07:38:37.838 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false
2023-05-21 07:38:37.839 [main] INFO JobContainer - DataX jobContainer starts job.
2023-05-21 07:38:37.840 [main] INFO JobContainer - Set jobId = 0
2023-05-21 07:38:38.368 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://172.22.22.67:52016;DatabaseName=test.
2023-05-21 07:38:38.463 [job-0] INFO OriginalConfPretreatmentUtil - table:[s] has columns:[a].
2023-05-21 07:38:38.484 [job-0] ERROR JobContainer - Exception when job run
java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
at com.alibaba.datax.common.util.Configuration.getList(Configuration.java:435) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.simplifyConf(OriginalConfPretreatmentUtil.java:70) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:34) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41) ~[plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.databendwriter.DatabendWriter$Job.init(DatabendWriter.java:36) ~[databendwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.start(Engine.java:86) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.entry(Engine.java:168) [datax-core-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.Engine.main(Engine.java:201) [datax-core-0.0.1-SNAPSHOT.jar:na]
2023-05-21 07:38:38.489 [job-0] INFO StandAloneJobContainerCommunicator - Total 0 records, 0 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2023-05-21 07:38:38.495 [job-0] ERROR Engine -

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Framework-02], Description:[DataX引擎运行过程出错,具体原因请参看DataX运行结束时的错误诊断信息 .]. - java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
at com.alibaba.datax.common.util.Configuration.getList(Configuration.java:435)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.simplifyConf(OriginalConfPretreatmentUtil.java:70)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:34)
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41)
at com.alibaba.datax.plugin.writer.databendwriter.DatabendWriter$Job.init(DatabendWriter.java:36)
at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
at com.alibaba.datax.core.Engine.start(Engine.java:86)
at com.alibaba.datax.core.Engine.entry(Engine.java:168)
at com.alibaba.datax.core.Engine.main(Engine.java:201)
- java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
at com.alibaba.datax.common.util.Configuration.getList(Configuration.java:435)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.simplifyConf(OriginalConfPretreatmentUtil.java:70)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:34)
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41)
at com.alibaba.datax.plugin.writer.databendwriter.DatabendWriter$Job.init(DatabendWriter.java:36)
at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
at com.alibaba.datax.core.Engine.start(Engine.java:86)
at com.alibaba.datax.core.Engine.entry(Engine.java:168)
at com.alibaba.datax.core.Engine.main(Engine.java:201)

at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:162)
at com.alibaba.datax.core.Engine.start(Engine.java:86)
at com.alibaba.datax.core.Engine.entry(Engine.java:168)
at com.alibaba.datax.core.Engine.main(Engine.java:201)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.List
at com.alibaba.datax.common.util.Configuration.getList(Configuration.java:435)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.simplifyConf(OriginalConfPretreatmentUtil.java:70)
at com.alibaba.datax.plugin.rdbms.writer.util.OriginalConfPretreatmentUtil.doPretreatment(OriginalConfPretreatmentUtil.java:34)
at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Job.init(CommonRdbmsWriter.java:41)
at com.alibaba.datax.plugin.writer.databendwriter.DatabendWriter$Job.init(DatabendWriter.java:36)
at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
... 3 more

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported configuration and the stack trace at Configuration.java:435 and OriginalConfPretreatmentUtil.java:70, then trace initialization through DatabendWriter.java:36. Run the shown DataX job with the SQL Server reader and Databend writer configuration; done means the job no longer fails during writer initialization with the reported ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.