alibaba / alibaba/DataX

datax3.0 otsstreamreader插件 状态表更新checkpoint问题

Open
#609 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

1. 通过otsstreamreader插件增量同步tablestore的增量数据
2. 设置stream的过期时间是72h
主要问题:
问题描述: otsstreamreader插件增量同步数据时会写一条数据,记录checkpoint到一张状态表中(TableStoreStreamReaderStatusTable),发现每次更新到状态表中的checkpoint都是一样的值,
所以同步几次后就会出现如下报错信息:
ps : 怀疑是每次同步数据时从状态表取checkpoint,而这个checkpoint一直没有变化(第一次写进去的值),72h后checkpoint在日志中找不到了,这个值就过期了,但是不知道问题怎么解决?
求帮忙!!!

2020-03-04 00:30:05.868 [pool-14-thread-1] ERROR RecordProcessor - Some fatal error has happened, shardId: e936e11e-1586-491b-ab7f-be0cf018279f_1568701257432681, LastShardIterator: null, NextShartIterator: HGV2ZW50X3JhbmdlXzE1Njg3MDEyNTc0MzI2ODE1ZTkzNmUxMWUtMTU4Ni00OTFiLWFiN2YtYmUwY2YwMTgyNzlmXzE1Njg3MDEyNTc0MzI2ODEcAgAAAAAAAACgOS4AAQAAAAEAAAD/////////f44=.
com.alicloud.openservices.tablestore.TableStoreException: Requested stream data is already trimmed or does not exist.
at com.alicloud.openservices.tablestore.core.CallbackImpledFuture.getResultWithoutLock(CallbackImpledFuture.java:107) ~[tablestore-4.2.0-20161107.070801-1.jar:na]
at com.alicloud.openservices.tablestore.core.CallbackImpledFuture.get(CallbackImpledFuture.java:78) ~[tablestore-4.2.0-20161107.070801-1.jar:na]
at com.alicloud.openservices.tablestore.SyncClient.waitForFuture(SyncClient.java:249) ~[tablestore-4.2.0-20161107.070801-1.jar:na]
at com.alicloud.openservices.tablestore.SyncClient.getStreamRecord(SyncClient.java:244) ~[tablestore-4.2.0-20161107.070801-1.jar:na]
at com.alibaba.datax.plugin.reader.otsstreamreader.internal.core.RecordProcessor.readAndProcessRecords(RecordProcessor.java:244) ~[otsstreamreader-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.reader.otsstreamreader.internal.core.RecordProcessor.run(RecordProcessor.java:149) ~[otsstreamreader-0.0.1-SNAPSHOT.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: com.alicloud.openservices.tablestore.TableStoreException: Requested stream data is already trimmed or does not exist.
at com.alicloud.openservices.tablestore.core.http.ErrorResponseHandler.handle(ErrorResponseHandler.java:56) ~[tablestore-4.2.0-20161107.070801-1.jar:na]

![1583734107969](https://user-images.githubusercontent.com/42470359/76187133-82665f00-620f-11ea-967f-bf621fa09419.jpg)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with RecordProcessor.java, especially readAndProcessRecords at line 244 and run at line 149, then trace how checkpoints are written to TableStoreStreamReaderStatusTable. Reproduce the repeated-checkpoint behavior with a 72-hour stream and inspect the TableStore getStreamRecord failure. Done means checkpoint updates advance correctly and the reader no longer resumes from trimmed stream data.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.