alibaba / alibaba/DataX

tsdbwriter不能正确写入

Open
#1,863 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

2023-08-01 09:59:00.175 [job-0] INFO JobContainer - jobContainer starts to do split ...
2023-08-01 09:59:00.175 [job-0] INFO JobContainer - Job set Channel-Number to 2 channels.
2023-08-01 09:59:00.176 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] splits to [1] tasks.
2023-08-01 09:59:00.177 [job-0] INFO JobContainer - DataX Writer.Job [tsdbwriter] splits to [1] tasks.
2023-08-01 09:59:00.199 [job-0] INFO JobContainer - jobContainer starts to do schedule ...
2023-08-01 09:59:00.201 [job-0] INFO JobContainer - Scheduler starts [1] taskGroups.
2023-08-01 09:59:00.203 [job-0] INFO JobContainer - Running by standalone Mode.
2023-08-01 09:59:00.206 [taskGroup-0] INFO TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-08-01 09:59:00.210 [taskGroup-0] INFO Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-08-01 09:59:00.210 [taskGroup-0] INFO Channel - Channel set record_speed_limit to -1, No tps activated.
2023-08-01 09:59:00.217 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-08-01 09:59:00.218 [0-0-0-writer] INFO TSDBConverter - columnName: ["device_id","time","data"], columnType: ["tag","timestamp","field_string"]
2023-08-01 09:59:00.219 [0-0-0-reader] INFO CommonRdbmsReader$Task - Begin to read record by Sql: [ select device_id,time,data from test;
] jdbcUrl:[jdbc:postgresql://xxxxx/fall_db].
2023-08-01 09:59:00.238 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [ select device_id,time,data from test;
] jdbcUrl:[jdbc:postgresql://xxxxx/fall_db].
2023-08-01 09:59:00.486 [0-0-0-writer] INFO TSDBWriter$Task - Task finished, write size: 1
2023-08-01 09:59:00.517 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[301]ms
2023-08-01 09:59:00.518 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-08-01 09:59:10.215 [job-0] INFO StandAloneJobContainerCommunicator - Total 1 records, 9 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-08-01 09:59:10.215 [job-0] INFO AbstractScheduler - Scheduler accomplished all tasks.
2023-08-01 09:59:10.216 [job-0] INFO JobContainer - DataX Writer.Job [tsdbwriter] do post work.
2023-08-01 09:59:10.216 [job-0] INFO JobContainer - DataX Reader.Job [postgresqlreader] do post work.
2023-08-01 09:59:10.216 [job-0] INFO JobContainer - DataX jobId [0] completed successfully.
2023-08-01 09:59:10.217 [job-0] INFO HookInvoker - No hook invoked, because base dir not exists or is a file: /root/dataX/datax/hook
2023-08-01 09:59:10.517 [pool-1-batch-put-thread-1] INFO BatchPutRunnable - The thread pool-1-batch-put-thread-1 is interrupted
2023-08-01 09:59:10.517 [pool-2-multi-field-batch-put-thread-1] INFO MultiFieldBatchPutRunnable - The thread pool-2-multi-field-batch-put-thread-1 is inter rupted
2023-08-01 09:59:10.518 [pool-3-points-batch-put-thread-1] INFO PointsCollectionPutRunnable - The thread pool-3-points-batch-put-thread-1 is interrupted. c ause null
2023-08-01 09:59:10.519 [job-0] INFO TSDBClient - The tsdb client has closed.
2023-08-01 09:59:10.520 [job-0] INFO JobContainer -
[total cpu info] =>
averageCpu | maxDeltaCpu | minDeltaCpu
-1.00% | -1.00% | -1.00%

[total gc info] =>
NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCT ime
PS MarkSweep | 1 | 1 | 1 | 0.028s | 0.028s | 0.028s
PS Scavenge | 1 | 1 | 1 | 0.013s | 0.013s | 0.013s

2023-08-01 09:59:10.520 [job-0] INFO JobContainer - PerfTrace not enable!
2023-08-01 09:59:10.521 [job-0] INFO StandAloneJobContainerCommunicator - Total 1 records, 9 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-08-01 09:59:10.522 [job-0] INFO JobContainer -
任务启动时刻 : 2023-08-01 09:58:59
任务结束时刻 : 2023-08-01 09:59:10
任务总计耗时 : 10s
任务平均流量 : 0B/s
记录写入速度 : 0rec/s
读出记录总数 : 1
读写失败总数 : 0

【2023-08-01 09:59:10.517 [pool-1-batch-put-thread-1] INFO BatchPutRunnable - The thread pool-1-batch-put-thread-1 is interrupted
2023-08-01 09:59:10.517 [pool-2-multi-field-batch-put-thread-1] INFO MultiFieldBatchPutRunnable - The thread pool-2-multi-field-batch-put-thread-1 is inter rupted
2023-08-01 09:59:10.518 [pool-3-points-batch-put-thread-1] INFO PointsCollectionPutRunnable - The thread pool-3-points-batch-put-thread-1 is interrupted. c ause null
2023-08-01 09:59:10.519 [job-0] INFO TSDBClient - The tsdb client has closed.】

postgres数据库导入lindorm时序引擎指定数据库和指定的表中,感觉batch线程中断了,不知道是什么原因?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the tsdbwriter entry point named in the logs, then trace TSDBWriter$Task, TSDBClient, and the BatchPutRunnable, MultiFieldBatchPutRunnable, and PointsCollectionPutRunnable threads. Reproduce the PostgreSQL-to-Lindorm write using the reported configuration and determine why the job reports success while the batch threads are interrupted; done means the destination table contains the source record and the interruption has an explained or corrected outcome.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.