mysql to hbase 任务正常启动,数据写入无异常,但是结束之后,hbase一条记录也没有
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
![Uploading image.png…]()
下面是日志,不知道问题出在哪里
2024-05-20 20:02:45 [BuildCommand.buildDataXParam-100] ------------------Command parameters:
2024-05-20 20:02:45 [ExecutorJobHandler.execute-57] ------------------DataX process id: 78670
2024-05-20 20:02:45 [ProcessCallbackThread.callbackLog-186]
----------- datax-web job callback finish.
2024-05-20 20:02:45 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:45 [AnalysisStatistics.analysisStatisticsLog-53] DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
2024-05-20 20:02:45 [AnalysisStatistics.analysisStatisticsLog-53] Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2024-05-20 20:02:45 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:45 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.421 [main] INFO VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.427 [main] INFO Engine - the machine info =>
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] osInfo: Red Hat, Inc. 1.8 25.402-b06
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] jvmInfo: Linux amd64 3.10.0-1160.114.2.el7.x86_64
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] cpu num: 16
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] totalPhysicalMemory: -0.00G
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] freePhysicalMemory: -0.00G
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] maxFileDescriptorCount: -1
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] currentOpenFileDescriptorCount: -1
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] GC Names [PS MarkSweep, PS Scavenge]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] MEMORY_NAME | allocation_size | init_size
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] PS Eden Space | 256.00MB | 256.00MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] Code Cache | 240.00MB | 2.44MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] Compressed Class Space | 1,024.00MB | 0.00MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] PS Survivor Space | 42.50MB | 42.50MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] PS Old Gen | 683.00MB | 683.00MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] Metaspace | -0.00MB | 0.00MB
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.444 [main] INFO Engine -
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "content":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "reader":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "name":"mysqlreader",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "column":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "`id`",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "`disease_name`",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "`disease_pic_path`"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "connection":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "jdbcUrl":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "jdbc:mysql://192.168.16.173:3306/bs"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "table":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "wheat_disease"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "password":"****",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "splitPk":"",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "username":"root"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] },
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "writer":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "name":"hbase11xwriter",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "parameter":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "column":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "index":1,
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "name":"0:disease_name",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] },
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "index":2,
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "name":"0:disease_pic_path",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "hbaseConfig":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "hbase.zookeeper.quorum":"hadoop01:2181"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] },
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "mode":"normal",
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "rowkeyColumn":[
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] {
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "index":0,
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "type":"string"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "table":"BS:WHEAT_DISEASE"
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] ],
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "setting":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "errorLimit":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "percentage":0.02,
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "record":0
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] },
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "speed":{
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "byte":1048576,
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] "channel":3
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] }
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.462 [main] WARN Engine - prioriy set to 0, because NumberFormatException, the value is: null
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.464 [main] INFO PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.464 [main] INFO JobContainer - DataX jobContainer starts job.
2024-05-20 20:02:46 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:46.465 [main] INFO JobContainer - Set jobId = 0
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.253 [job-0] INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://192.168.16.173:3306/bs?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.521 [job-0] INFO OriginalConfPretreatmentUtil - table:[wheat_disease] has columns:[id,disease_name,disease_pic_path].
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.532 [job-0] INFO JobContainer - jobContainer starts to do prepare ...
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.533 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do prepare work .
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.533 [job-0] INFO JobContainer - DataX Writer.Job [hbase11xwriter] do prepare work .
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.533 [job-0] INFO JobContainer - jobContainer starts to do split ...
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.534 [job-0] INFO JobContainer - Job set Max-Byte-Speed to 1048576 bytes.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.538 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] splits to [1] tasks.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.538 [job-0] INFO JobContainer - DataX Writer.Job [hbase11xwriter] splits to [1] tasks.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.549 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.551 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.553 [job-0] INFO JobContainer - Running by standalone Mode.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.559 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.566 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.566 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.577 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.580 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [select `id`,`disease_name`,`disease_pic_path` from wheat_disease
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] ] jdbcUrl:[jdbc:mysql://192.168.16.173:3306/bs?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:47.616 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select `id`,`disease_name`,`disease_pic_path` from wheat_disease
2024-05-20 20:02:47 [AnalysisStatistics.analysisStatisticsLog-53] ] jdbcUrl:[jdbc:mysql://192.168.16.173:3306/bs?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true].
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.117 [0-0-0-writer] INFO ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.118 [0-0-0-writer] INFO ZooKeeper - Client environment:host.name=hadoop01
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.118 [0-0-0-writer] INFO ZooKeeper - Client environment:java.version=1.8.0_402
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.118 [0-0-0-writer] INFO ZooKeeper - Client environment:java.vendor=Red Hat, Inc.
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.118 [0-0-0-writer] INFO ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.402.b06-1.el7_9.x86_64/jre
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.118 [0-0-0-writer] INFO ZooKeeper - Client environment:java.class.path=/export/servers/datax/lib/._commons-logging-1.1.1.jar:/export/servers/datax/lib/commons-logging-1.1.1.jar:/export/servers/datax/lib/._httpcore-4.4.jar:/export/servers/datax/lib/httpcore-4.4.jar:/export/servers/datax/lib/._fastjson-1.1.46.sec01.jar:/export/servers/datax/lib/fastjson-1.1.46.sec01.jar:/export/servers/datax/lib/._commons-configuration-1.10.jar:/export/servers/datax/lib/commons-configuration-1.10.jar:/export/servers/datax/lib/._httpclient-4.4.jar:/export/servers/datax/lib/httpclient-4.4.jar:/export/servers/datax/lib/._commons-cli-1.2.jar:/export/servers/datax/lib/commons-cli-1.2.jar:/export/servers/datax/lib/._fluent-hc-4.4.jar:/export/servers/datax/lib/fluent-hc-4.4.jar:/export/servers/datax/lib/._commons-lang3-3.3.2.jar:/export/servers/datax/lib/commons-lang3-3.3.2.jar:/export/servers/datax/lib/._janino-2.5.16.jar:/export/servers/datax/lib/janino-2.5.16.jar:/export/servers/datax/lib/._commons-codec-1.9.jar:/export/servers/datax/lib/commons-codec-1.9.jar:/export/servers/datax/lib/._logback-core-1.0.13.jar:/export/servers/datax/lib/logback-core-1.0.13.jar:/export/servers/datax/lib/._groovy-all-2.1.9.jar:/export/servers/datax/lib/groovy-all-2.1.9.jar:/export/servers/datax/lib/._commons-io-2.4.jar:/export/servers/datax/lib/commons-io-2.4.jar:/export/servers/datax/lib/._datax-common-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/datax-common-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/._hamcrest-core-1.3.jar:/export/servers/datax/lib/hamcrest-core-1.3.jar:/export/servers/datax/lib/._commons-collections-3.2.1.jar:/export/servers/datax/lib/commons-collections-3.2.1.jar:/export/servers/datax/lib/._datax-transformer-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/._commons-lang-2.6.jar:/export/servers/datax/lib/commons-lang-2.6.jar:/export/servers/datax/lib/._commons-beanutils-1.9.2.jar:/export/servers/datax/lib/commons-beanutils-1.9.2.jar:/export/servers/datax/lib/._logback-classic-1.0.13.jar:/export/servers/datax/lib/logback-classic-1.0.13.jar:/export/servers/datax/lib/._commons-math3-3.1.1.jar:/export/servers/datax/lib/commons-math3-3.1.1.jar:/export/servers/datax/lib/._datax-core-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/datax-core-0.0.1-SNAPSHOT.jar:/export/servers/datax/lib/._slf4j-api-1.7.10.jar:/export/servers/datax/lib/slf4j-api-1.7.10.jar:.
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:java.io.tmpdir=/tmp
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:java.compiler=
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:os.name=Linux
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:os.arch=amd64
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:os.version=3.10.0-1160.114.2.el7.x86_64
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:user.name=root
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:user.home=/root
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.119 [0-0-0-writer] INFO ZooKeeper - Client environment:user.dir=/export/servers/datax-web-2.1.2/bin
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.120 [0-0-0-writer] INFO ZooKeeper - Initiating client connection, connectString=hadoop01:2181 sessionTimeout=90000 watcher=hconnection-0x68497ed90x0, quorum=hadoop01:2181, baseZNode=/hbase
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.139 [0-0-0-writer-SendThread(hadoop01:2181)] INFO ClientCnxn - Opening socket connection to server hadoop01/192.168.16.130:2181. Will not attempt to authenticate using SASL (unknown error)
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.143 [0-0-0-writer-SendThread(hadoop01:2181)] INFO ClientCnxn - Socket connection established to hadoop01/192.168.16.130:2181, initiating session
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.151 [0-0-0-writer-SendThread(hadoop01:2181)] INFO ClientCnxn - Session establishment complete on server hadoop01/192.168.16.130:2181, sessionid = 0xff8f89d7afb6003e, negotiated timeout = 40000
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.682 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1108]ms
2024-05-20 20:02:48 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:48.682 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.575 [job-0] INFO StandAloneJobContainerCommunicator - Total 8 records, 700 bytes | Speed 70B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.576 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.576 [job-0] INFO JobContainer - DataX Writer.Job [hbase11xwriter] do post work.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.576 [job-0] INFO JobContainer - DataX Reader.Job [mysqlreader] do post work.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.576 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.577 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /export/servers/datax/hook
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.577 [job-0] INFO JobContainer -
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] [total cpu info] =>
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] averageCpu | maxDeltaCpu | minDeltaCpu
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] -1.00% | -1.00% | -1.00%
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] [total gc info] =>
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTime
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] PS MarkSweep | 1 | 1 | 1 | 0.162s | 0.162s | 0.162s
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] PS Scavenge | 1 | 1 | 1 | 0.060s | 0.060s | 0.060s
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.577 [job-0] INFO JobContainer - PerfTrace not enable!
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.578 [job-0] INFO StandAloneJobContainerCommunicator - Total 8 records, 700 bytes | Speed 70B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 2024-05-20 20:02:57.579 [job-0] INFO JobContainer -
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 任务启动时刻 : 2024-05-20 20:02:46
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 任务结束时刻 : 2024-05-20 20:02:57
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 任务总计耗时 : 11s
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 任务平均流量 : 70B/s
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 记录写入速度 : 0rec/s
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 读出记录总数 : 8
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] 读写失败总数 : 0
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53]
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] May 20, 2024 8:02:47 PM org.apache.hadoop.util.NativeCodeLoader
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] May 20, 2024 8:02:48 PM org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper
2024-05-20 20:02:57 [AnalysisStatistics.analysisStatisticsLog-53] INFO: Process identifier=hconnection-0x68497ed9 connecting to ZooKeeper ensemble=hadoop01:2181
2024-05-20 20:02:57 [JobThread.run-165]
----------- datax-web job execute end(finish) -----------
----------- ReturnT:ReturnT [code=200, msg=LogStatistics{taskStartTime=2024-05-20 20:02:46, taskEndTime=2024-05-20 20:02:57, taskTotalTime=11s, taskAverageFlow=70B/s, taskRecordWritingSpeed=0rec/s, taskRecordReaderNum=8, taskRecordWriteFailNum=0}, content=null]
2024-05-20 20:02:58 [TriggerCallbackThread.callbackLog-186]
----------- datax-web job callback finish.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the logged mysqlreader-to-hbase11xwriter job with the shown column indexes, rowkeyColumn, table, and hbaseConfig, then inspect the task output and HBase table state. Done means identifying why the completed job writes no records and confirming that records appear in BS:WHEAT_DISEASE after the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100