[TableCopy] Controller crash recovery for in-progress table replication jobs
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
## Background
During a table copy operation (PR #17521), the replication job runs on the controller that received the API request. If that controller crashes or restarts mid-job, the in-progress job is lost and does not automatically resume.
## Current behavior
Users need to delete the target table and restart the copy from the beginning.
## Proposed improvements
**Short-term (manual resume API):**
- Provide a `/resume` API that cancels the job on the old controller, computes the remaining segments to copy (source segments − segments already above the watermark − segments already in target), and re-triggers the copy.
**Long-term (automatic recovery by lead controller):**
- The cluster's lead controller runs a background scan thread:
1. Scans all table replication controller jobs.
2. If the controller that owns a job UUID is dead, select a new controller and invoke the resume endpoint.
3. If the controller is alive but no longer has the job in its local executor, send a Helix message to the assigned controller; if the controller notices it owns the job UUID but has no running task, recreate the copy tasks.
## Related
- PR: https://github.com/apache/pinot/pull/17521
Contributor guide
Research direction
Start by reading PR #17521 and tracing the table-copy replication job flow described in this issue, including the resume API, controller ownership, local executor, and Helix message paths. Done means an in-progress job can resume after its controller crashes or loses the task, without requiring users to delete the target table and restart from the beginning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100