scylladb / scylladb/scylla-cdc-java
StackOverflowError when retrying reading from the middle of window with large amount of data
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 32
- Forks
- 23
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 3
Description
When there is an exception while reading a window, we restart it with TaskState pointing to last correctly read change. After restarting, we read the entire window from beginning, but we ignore unnecessary rows (up to last correctly read change).
The problem with current implementation is that it does this ignoring recursively (see last findNext at line 148):
https://github.com/scylladb/scylla-cdc-java/blob/a2c3c1823d1f5c210bd210e2d048de0f02ceeb93/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3WorkerCQL.java#L104-L151
I have observed one instance of this causing StackOverflowError. What's bad is that it won't be able to succeed upon restart (it will try to do it again after restart and fail in the same fashion).
The long term fix for this is to implement smarter resuming of reading (do not skip client-side, but craft such CQL queries that will read the correct portion of window).
/cc @haaawk Would you approve of a quick fix for this problem (wrapping findNext with while to do the skipping iteratively, not recursively) - doing "smarter resuming" will require more time and there are other priorities right now. Of course I will do "smarter resuming" later.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3WorkerCQL.java, examining the findNext logic around lines 104-151 and how retry state skips already-read changes. Confirm that skipping a large number of rows no longer relies on recursive calls and that retrying a window can complete without StackOverflowError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100