scylladb / scylladb/scylla-cdc-java

StackOverflowError when retrying reading from the middle of window with large amount of data

Open
#36 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

base
Dominant language
Java
Stars
32
Forks
23
Avg merge
2d 6h
Merged PRs (30d)
3

Description

When there is an exception while reading a window, we restart it with TaskState pointing to last correctly read change. After restarting, we read the entire window from beginning, but we ignore unnecessary rows (up to last correctly read change).

The problem with current implementation is that it does this ignoring recursively (see last findNext at line 148):
https://github.com/scylladb/scylla-cdc-java/blob/a2c3c1823d1f5c210bd210e2d048de0f02ceeb93/scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3WorkerCQL.java#L104-L151

I have observed one instance of this causing StackOverflowError. What's bad is that it won't be able to succeed upon restart (it will try to do it again after restart and fail in the same fashion).

The long term fix for this is to implement smarter resuming of reading (do not skip client-side, but craft such CQL queries that will read the correct portion of window).

/cc @haaawk Would you approve of a quick fix for this problem (wrapping findNext with while to do the skipping iteratively, not recursively) - doing "smarter resuming" will require more time and there are other priorities right now. Of course I will do "smarter resuming" later.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in scylla-cdc-driver3/src/main/java/com/scylladb/cdc/cql/driver3/Driver3WorkerCQL.java, examining the findNext logic around lines 104-151 and how retry state skips already-read changes. Confirm that skipping a large number of rows no longer relies on recursive calls and that retrying a window can complete without StackOverflowError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.