apache / apache/pinot

[TableCopy] Controller crash recovery for in-progress table replication jobs

Open
#17,933 1 comment 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

## Background

During a table copy operation (PR #17521), the replication job runs on the controller that received the API request. If that controller crashes or restarts mid-job, the in-progress job is lost and does not automatically resume.

## Current behavior

Users need to delete the target table and restart the copy from the beginning.

## Proposed improvements

**Short-term (manual resume API):**
- Provide a `/resume` API that cancels the job on the old controller, computes the remaining segments to copy (source segments − segments already above the watermark − segments already in target), and re-triggers the copy.

**Long-term (automatic recovery by lead controller):**
- The cluster's lead controller runs a background scan thread:
1. Scans all table replication controller jobs.
2. If the controller that owns a job UUID is dead, select a new controller and invoke the resume endpoint.
3. If the controller is alive but no longer has the job in its local executor, send a Helix message to the assigned controller; if the controller notices it owns the job UUID but has no running task, recreate the copy tasks.

## Related

- PR: https://github.com/apache/pinot/pull/17521

Contributor guide

Open the contributing guide

Research direction

Start by reading PR #17521 and tracing the table-copy replication job flow described in this issue, including the resume API, controller ownership, local executor, and Helix message paths. Done means an in-progress job can resume after its controller crashes or loses the task, without requiring users to delete the target table and restart from the beginning.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.