doris writer 在数据中含有%时，无法迁移数据

Open

#2,079 opened on Mar 20, 2024 0 comments 0 reactions 0 assignees View on GitHub

Dominant language: Java
Stars: 17.4k
Forks: 5.7k
PR merge metrics: No merged PRs in 30d

Description

我的表以及数据：
CREATE TABLE table_0 (column_1int NULL DEFAULT NULL,column_2` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL DEFAULT NULL
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_0900_ai_ci ROW_FORMAT = Dynamic;

-- Records of table_0

INSERT INTO table_0 VALUES (1, '刘禹福20%');
INSERT INTO table_0 VALUES (2, '王许前80%');
`

我的mysql2doris.json文件：
{ "core": { "transport": { "channel": { "speed": { "byte": 1048576 } } } }, "job": { "setting": { "speed": { "channel": 4, "byte": 1048576 }, "errorLimit": { "record": 0, "percentage": 0.02 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "test", "password": "**********", "column": [ "column_1", "column_2" ], "where": "1=1", "splitPk": "", "connection": [ { "table": [ "test.table_0" ], "jdbcUrl": [ "jdbc:mysql://172.16.xx.xx:3306/test?useSSL=false" ] } ] } }, "writer": { "name": "doriswriter", "parameter": { "username": "test", "password": "**********", "column": [ "column_1", "column_2" ], "loadUrl": [ "172.16.xx.xx:8030" ], "loadProps": { "format": "json" }, "connection": [ { "jdbcUrl": "jdbc:mysql://172.16.xx.xx:9030/testx?useSSL=false", "selectedDatabase": "testx", "table": [ "table_0" ] } ] } } } ] } }

运行之后报错的日志：
`
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2024-03-20 10:56:47.501 [main] INFO MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2024-03-20 10:56:47.504 [main] INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2024-03-20 10:56:47.515 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-03-20 10:56:47.520 [main] INFO Engine - the machine info =>

osInfo: Huawei Technologies Co., Ltd 1.8 25.242-b08
jvmInfo: Linux amd64 4.19.90-23.8.v2101.ky10.x86_64
cpu num: 32

totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1

GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME | allocation_size | init_size
PS Eden Space | 256.00MB | 256.00MB
Code Cache | 240.00MB | 2.44MB
Compressed Class Space | 1,024.00MB | 0.00MB
PS Survivor Space | 42.50MB | 42.50MB
PS Old Gen | 683.00MB | 683.00MB
Metaspace | -0.00MB | 0.00MB
2024-03-20 10:56:47.537 [main] INFO Engine -
{
"content":[
{
"reader":{
"name":"mysqlreader",
"parameter":{
"column":[
"column_1",
"column_2"
],
"connection":[
{
"jdbcUrl":[
"jdbc:mysql://172.16.xx.xx:3306/test?useSSL=false"
],
"table":[
"test.table_0"
]
}
],
"password":"******",
"splitPk":"",
"username":"test",
"where":"1=1"
}
},
"writer":{
"name":"doriswriter",
"parameter":{
"column":[
"column_1",
"column_2"
],
"connection":[
{
"jdbcUrl":"jdbc:mysql://172.16.xx.xx:9030/test?useSSL=false",
"selectedDatabase":"test",
"table":[
"table_0"
]
}
],
"loadProps":{
"format":"json"
},
"loadUrl":[
"172.16.xx.xx:8030"
],
"password":"",
"username":"test"
}
}
}
],
"setting":{
"errorLimit":{
"percentage":0.02,
"record":0
},
"speed":{
"byte":33554432,
"channel":32
}
}
}

2024-03-20 10:56:47.551 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2024-03-20 10:56:47.553 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-03-20 10:56:47.553 [main] INFO JobContainer - DataX jobContainer starts job.
2024-03-20 10:56:47.554 [main] INFO JobContainer - Set jobId = 0
2024-03-20 10:56:47.844 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://172.16.xx.xx:3306/test?useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2024-03-20 10:56:47.857 [job-0] INFO OriginalConfPretreatmentUtil - table:[test.table_0] has columns:[column_1,column_2].
2024-03-20 10:56:47.891 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2024-03-20 10:56:47.892 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2024-03-20 10:56:47.892 [job-0] INFO JobContainer - DataX Writer.Job [doriswriter] do prepare work .
2024-03-20 10:56:47.893 [job-0] INFO JobContainer - jobContainer starts to do split ...
2024-03-20 10:56:47.894 [job-0] INFO JobContainer - Job set Max-Byte-Speed to 33554432 bytes.
2024-03-20 10:56:47.896 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2024-03-20 10:56:47.897 [job-0] INFO JobContainer - DataX Writer.Job [doriswriter] splits to [1] tasks.
2024-03-20 10:56:47.912 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2024-03-20 10:56:47.915 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-03-20 10:56:47.917 [job-0] INFO JobContainer - Running by standalone Mode.
2024-03-20 10:56:47.921 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2024-03-20 10:56:47.925 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to https://github.com/apache/doris/commit/1048576643ecf0046dcb770f33826bd01ac99405.
2024-03-20 10:56:47.925 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-03-20 10:56:47.939 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2024-03-20 10:56:47.943 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select column_1,column_2 from test.table_0 where (1=1)
] jdbcUrl:[jdbc:mysql://172.16.xx.xx:3306/test?useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2024-03-20 10:56:47.959 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select column_1,column_2 from test.table_0 where (1=1)
] jdbcUrl:[jdbc:mysql://172.16.xx.xx:3306/test?useSSL=false&yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2024-03-20 10:56:47.978 [Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[2] bytes[84] label[datax_doris_writer_0078661c-78d0-40e7-81ce-3f956580158f].
2024-03-20 10:56:47.978 [Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://172.16.xx.xx:8030/api/test/table_0/_stream_load', size: '87'
2024-03-20 10:56:48.187 [Thread-1] WARN DorisStreamLoadObserver - Request failed with code:500
2024-03-20 10:56:48.188 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad response :null
2024-03-20 10:56:48.191 [Thread-1] WARN DorisWriterManager - Failed to flush batch data to Doris, retry times = 0
java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2024-03-20 10:56:49.194 [Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[2] bytes[84] label[datax_doris_writer_0078661c-78d0-40e7-81ce-3f956580158f].
2024-03-20 10:56:49.194 [Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://172.16.xx.xx:8030/api/test/table_0/_stream_load', size: '87'
2024-03-20 10:56:49.204 [Thread-1] WARN DorisStreamLoadObserver - Request failed with code:500
2024-03-20 10:56:49.204 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad response :null
2024-03-20 10:56:49.204 [Thread-1] WARN DorisWriterManager - Failed to flush batch data to Doris, retry times = 1
java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2024-03-20 10:56:51.207 [Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[2] bytes[84] label[datax_doris_writer_0078661c-78d0-40e7-81ce-3f956580158f].
2024-03-20 10:56:51.207 [Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://172.16.xx.xx:8030/api/test/table_0/_stream_load', size: '87'
2024-03-20 10:56:51.217 [Thread-1] WARN DorisStreamLoadObserver - Request failed with code:500
2024-03-20 10:56:51.218 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad response :null
2024-03-20 10:56:51.218 [Thread-1] WARN DorisWriterManager - Failed to flush batch data to Doris, retry times = 2
java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2024-03-20 10:56:54.221 [Thread-1] INFO DorisStreamLoadObserver - Start to join batch data: rows[2] bytes[84] label[datax_doris_writer_0078661c-78d0-40e7-81ce-3f956580158f].
2024-03-20 10:56:54.221 [Thread-1] INFO DorisStreamLoadObserver - Executing stream load to: 'http://172.16.xx.xx:8030/api/test/table_0/_stream_load', size: '87'
2024-03-20 10:56:54.232 [Thread-1] WARN DorisStreamLoadObserver - Request failed with code:500
2024-03-20 10:56:54.232 [Thread-1] INFO DorisStreamLoadObserver - StreamLoad response :null
2024-03-20 10:56:54.233 [Thread-1] WARN DorisWriterManager - Failed to flush batch data to Doris, retry times = 3
java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134) [doriswriter-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
2024-03-20 10:56:54.234 [0-0-0-writer] ERROR WriterRunner - Writer Runner Received Exceptions:
com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more

java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more

at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48) ~[datax-common-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:152) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65) ~[datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_242]
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
... 2 common frames omitted
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
... 3 common frames omitted
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
... 1 common frames omitted
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163) ~[doriswriter-0.0.1-SNAPSHOT.jar:na]
... 3 common frames omitted
Exception in thread "taskGroup-0" com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more

at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:48)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:152)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
... 2 more
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more
2024-03-20 10:56:57.931 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 14 bytes | Speed 1B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 0.00%
2024-03-20 10:56:57.932 [job-0] ERROR JobContainer - 运行scheduler 模式[standalone]出错.
2024-03-20 10:56:57.932 [job-0] ERROR JobContainer - Exception when job run
com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more

经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[DBUtilErrorCode-05], Description:[往您配置的写入表中写入数据时失败.]. - java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:113)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriter$Task.post(DorisWriter.java:150)
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:65)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to Doris failed.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.checkFlushException(DorisWriterManager.java:189)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.waitAsyncFlushingDone(DorisWriterManager.java:150)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.flush(DorisWriterManager.java:98)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.close(DorisWriterManager.java:111)
... 3 more
Caused by: java.io.IOException: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:170)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.access$000(DorisWriterManager.java:19)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager$1.run(DorisWriterManager.java:134)
... 1 more
Caused by: java.io.IOException: Unable to flush data to Doris: unknown result status.
at com.alibaba.datax.plugin.writer.doriswriter.DorisStreamLoadObserver.streamLoad(DorisStreamLoadObserver.java:66)
at com.alibaba.datax.plugin.writer.doriswriter.DorisWriterManager.asyncFlush(DorisWriterManager.java:163)
... 3 more

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the provided mysql2doris configuration and reproduce the failure using values containing %. Then inspect DorisStreamLoadObserver.java around line 66 and DorisWriterManager.java around lines 134-163; done means the same records migrate successfully without the HTTP 500 and retry errors.

Written by the indexing model from the issue text.

Assessment

Tech stack: java, mysql
Domain: data-engineering, databases
Issue type: Bug
Difficulty: 3/5
Estimated time: 1-2 days
Activity status: Stale
Clarity: Mostly clear
Newbie friendliness: 45/100

doris writer 在数据中含有%时，无法迁移数据

Description

Contributor guide

Research direction

Assessment

Get new issues in your inbox