scylladb / scylladb/scylla-cdc-java

Don't restart generation reading from the beginning on generation table query failure

Open
#40 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

base
Dominant language
Java
Stars
32
Forks
23
Avg merge
2d 6h
Merged PRs (30d)
3

Description

Currently the Master process, whenever it gets an exception (any exception), restarts reading the last generation from the beginning.

But that doesn't make much sense from UX point of view.

Suppose that the last generation was created a week ago.
Now a transient failure (a network partition) causes the Master to fail to fetch the list of generations (which it does periodically to check if there are new generations). This causes an exception, which causes Master to create completely new workers which read that generation from the start! So we have to process a week of work right from the beginning! A week is just an example, this could be a month or longer!

If the Master successfully created workers for the last generation, it shouldn't recreate them on the first opportunity. Let them continue their work (but monitor them for crashes etc.). Probably the only situation where Master should recreate workers would be when it failed to create all workers in the first place; in this case it should destroy the existing workers and try again.

cc @haaawk @avelanarius

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the Master process path that periodically queries the generation table, handles exceptions, and creates or recreates workers. Reproduce a transient query failure and verify that existing workers continue processing; worker recreation should occur only when initial worker creation did not complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.