[Tracking] CCR (Cross-Cluster Replication): issues, features & bug-fix progress

## Background

Doris **Cross-Cluster Replication (CCR)** keeps a target cluster in sync with a source cluster: full initialization via `BACKUP`/`RESTORE`, then continuous incremental sync by pulling binlog commit sequences from the source and applying them on the target through transactional `ingest binlog`. The open-source syncer lives in [`selectdb/ccr-syncer`](https://github.com/selectdb/ccr-syncer); Doris FE/BE provide backup/restore, binlog, and ingest APIs.

This is an **umbrella / tracking issue** to consolidate CCR-related work — bug fixes, feature requests, and full-sync performance improvements — across **apache/doris** and **selectdb/ccr-syncer**, so contributors and maintainers have a single place to see status and pick up work. Related umbrella: **#66252**. Please comment below to add items or volunteer.

---

## Open bugs & compatibility breakage

- [ ] **#67120** — Shared-Nothing CCR `IngestBinlog` may ignore MOW delete bitmap calculation failures, commit the rowset, and expose duplicate unique keys.
- [ ] **#66571** — `swap=false REPLACE TABLE` may skip the `REPLACE_TABLE` binlog on the master FE when only table-level binlog is enabled and `BinlogConfigCache` is cold. *(#48919 fixes only the DB-level-enabled case; a full fix is proposed in [HYDCP/hy-doris#76](https://github.com/HYDCP/hy-doris/pull/76))*
- [ ] **selectdb/ccr-syncer#654** — Table-level CCR silently ignores upstream `DROP TABLE`; task keeps polling with no warning. → fix in **selectdb/ccr-syncer#660** *(open, lint CI failing — needs review)*
- [ ] **selectdb/ccr-syncer#677** — Table-level CCR may miss table-id switch after `REPLACE TABLE … PROPERTIES('swap'='true')`; `REPLACE_TABLE` binlog `tableIds` only attaches `origTblId`. *(no Doris PR yet — needs `BinlogManager.addReplaceTable` fix)*
- [ ] **#61063** — `get_binlog_info` returns not found for existing binlog files after tablet clone; CCR incremental ingest fails. *(no PR yet)*
- [ ] **selectdb/ccr-syncer#648** — Deleting a job and immediately recreating one with the same name can corrupt or wipe task metadata. → fixed in **selectdb/ccr-syncer#659** *(merged 2025-12 — issue still open, pending verification & close)*
- [ ] **#65632** — FE backup staging directories under `tmp_dir` may not be cleaned up after backup jobs. → fix in **#65642** *(open, needs review)* · also in [HYDCP/hy-doris#66](https://github.com/HYDCP/hy-doris/pull/66), [#67](https://github.com/HYDCP/hy-doris/pull/67) *(not yet in apache/doris)*

## Open feature requests

### Storage medium sync

Cross-cluster clusters often differ in SSD/HDD policy; restore and `MODIFY PARTITION` need explicit medium control on both sides.

- [ ] **#58897** — Enhance restore storage medium control. *(open, needs review — large PR)*
- [ ] **#63011** — Support medium allocation mode (part 1 of #58897 series). *(open, needs review)*
- [ ] **selectdb/ccr-syncer#624** — Medium sync policy in CCR tasks (CREATE TABLE retry + MODIFY PARTITION medium sync). *(draft, conflicting — blocked on #58897 / #63011)*

### Replication-num heterogeneity

- [x] **selectdb/ccr-syncer#649** — Custom `replication_num` for downstream cluster. *(merged)*
- [x] **selectdb/ccr-syncer#658** — Legacy job compatibility for `replication_num`. *(merged)*
- [x] **selectdb/ccr-syncer#651** — Remove BE caching so restore sees current backend set after scale-out. *(merged)*
- [ ] Allow `atomic_restore` with different replica num. → [HYDCP/hy-doris#51](https://github.com/HYDCP/hy-doris/pull/51) *(not yet in apache/doris)*

### CCR full-sync optimization

CCR full sync is `BACKUP → RESTORE`; at scale FE memory, concurrency limits, and staging cleanup become bottlenecks.

- [ ] Reduce FE memory usage of backup jobs with massive tablets (Gson DOM serialization, giant String materialization, double journal serialization). → [HYDCP/hy-doris#63](https://github.com/HYDCP/hy-doris/pull/63) *(open, needs review — not yet in apache/doris)*
- [ ] **#65321** — Strip replica info from backup meta to shrink backup metadata and reduce FE heap pressure. *(draft — split out from the backup FE memory work above; restore recreates replicas anyway, so serialized Replica objects are redundant)*
- [ ] **#64492** — Streaming Gson adapter for RestoreJob Guava Table fields (restore-side memory). *(closed without merge — re-submit needed)*
- [ ] Streaming Gson adapter for RestoreJob Guava Table fields (restore-side memory). → [HYDCP/hy-doris#49](https://github.com/HYDCP/hy-doris/pull/49) *(not yet in apache/doris)*
- [ ] **#61710** — Support concurrent table-level backup/restore. *(open, needs review)*
- [x] **#59580** — Preserve cross-database references when restoring views. *(merged)*
- [x] **#63831** — Handle incomplete dynamic partition properties on restore. *(merged)*
- [ ] Clean backup local job dir after completion. → [HYDCP/hy-doris#66](https://github.com/HYDCP/hy-doris/pull/66) *(not yet in apache/doris)*
- [ ] Centralize backup staging directory cleanup on checkpoint replay and job eviction. → [HYDCP/hy-doris#67](https://github.com/HYDCP/hy-doris/pull/67) *(not yet in apache/doris)*

### Single-replica ingest binlog

- [ ] Single-replica ingest binlog for 1-replica target clusters (paired FE/BE + syncer changes). *(no upstream PRs yet)*

### CCR ops & FE connectivity

- [x] **selectdb/ccr-syncer#645** — JobCollector RPC connection reuse. *(merged — closes selectdb/ccr-syncer#644)*
- [x] **selectdb/ccr-syncer#653** — Fix `/job_status` not updating after Pause/Resume. *(merged)*
- [ ] Return valid master address for `NOT_MASTER` errors. → [HYDCP/hy-doris#53](https://github.com/HYDCP/hy-doris/pull/53) *(not yet in apache/doris)*
- [ ] Return master address for snapshot RPCs. → [HYDCP/hy-doris#59](https://github.com/HYDCP/hy-doris/pull/59) *(not yet in apache/doris)*

## Open pull requests awaiting review

| PR | Repo | Summary | Closes |
|----|------|---------|--------|
| **selectdb/ccr-syncer#660** | ccr-syncer | Auto-recover table sync when upstream table is recreated | selectdb/ccr-syncer#654 |
| **selectdb/ccr-syncer#624** | ccr-syncer | Medium sync policy in CCR tasks *(draft)* | — |
| **#65642** | doris | Clean up FE backup staging directories | #65632 |
| **#65321** | doris | Strip replica info from backup meta *(draft; part of backup FE memory optimization)* | — |
| **[HYDCP/hy-doris#63](https://github.com/HYDCP/hy-doris/pull/63)** | hy-doris | Reduce FE memory for backup jobs with massive tablets | — |
| **#58897** | doris | Enhance restore storage medium control | — |
| **#63011** | doris | Support medium allocation mode | — |
| **#61710** | doris | Support concurrent backup/restore | — |

---

## Recently completed (last ~12 months)

### ccr-syncer

- [x] **selectdb/ccr-syncer#676** — Fix nil panic when transaction becomes visible.
- [x] **selectdb/ccr-syncer#672** — Persist `Dest.TableId` after `partialSync` in TableSync mode.
- [x] **selectdb/ccr-syncer#659** — Optimize job deletion to prevent same-name recreate races. Closes **selectdb/ccr-syncer#648** *(pending issue close)*.
- [x] **selectdb/ccr-syncer#658** — Legacy job compatibility for `replication_num`.
- [x] **selectdb/ccr-syncer#653** — Fix `/job_status` not updating after Pause/Resume.
- [x] **selectdb/ccr-syncer#651** — Remove BE caching; fix restore with stale backend info after scale-out.
- [x] **selectdb/ccr-syncer#649** — Support custom `replication_num` for downstream cluster.
- [x] **selectdb/ccr-syncer#645** — JobCollector RPC connection reuse. Closes **selectdb/ccr-syncer#644**.
- [x] **selectdb/ccr-syncer#638** — `tableRecord` add delta rows.
- [x] **selectdb/ccr-syncer#486** — Handle dummy binlog.
- [x] **selectdb/ccr-syncer#662** (w41ter) — Fix INSERT OVERWRITE in TableSync mode. Closes **selectdb/ccr-syncer#664**.

### apache/doris

- [x] **#59580** — Preserve cross-database references when restoring views.
- [x] **#63831** — Handle incomplete dynamic partition properties on restore.

Earlier / closed without merge (for reference)

- **#59044** — Detect dropped tables early in binlog API *(stale-closed Jun 2026; referenced by selectdb/ccr-syncer#660)*
- **#64492** — RestoreJob Gson streaming *(closed without merge — see [HYDCP/hy-doris#49](https://github.com/HYDCP/hy-doris/pull/49))*
- **#52391** / **#53120** — Earlier storage-medium attempts, superseded by #58897 / #63011
- **selectdb/ccr-syncer#647** — `atomic_restore` replica mismatch discussion → doris#42337 / [HYDCP/hy-doris#51](https://github.com/HYDCP/hy-doris/pull/51)
- **selectdb/ccr-syncer#652** / **#374** — Cross-version feasibility & getBinlog Q&A *(closed)*

This is an umbrella issue covering Apache Doris FE/BE CCR work and the external ccr-syncer repository, rather than one scoped change. Start by choosing one unchecked item and reading its linked issue or pull request; done means completing that item and updating its status here.

Written by the indexing model from the issue text.