scylladb / scylladb/scylla-cdc-java
Don't restart generation reading from the beginning on generation table query failure
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 32
- Forks
- 23
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 3
Description
Currently the Master process, whenever it gets an exception (any exception), restarts reading the last generation from the beginning.
But that doesn't make much sense from UX point of view.
Suppose that the last generation was created a week ago.
Now a transient failure (a network partition) causes the Master to fail to fetch the list of generations (which it does periodically to check if there are new generations). This causes an exception, which causes Master to create completely new workers which read that generation from the start! So we have to process a week of work right from the beginning! A week is just an example, this could be a month or longer!
If the Master successfully created workers for the last generation, it shouldn't recreate them on the first opportunity. Let them continue their work (but monitor them for crashes etc.). Probably the only situation where Master should recreate workers would be when it failed to create all workers in the first place; in this case it should destroy the existing workers and try again.
cc @haaawk @avelanarius
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the Master process path that periodically queries the generation table, handles exceptions, and creates or recreates workers. Reproduce a transient query failure and verify that existing workers continue processing; worker recreation should occur only when initial worker creation did not complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100