[Bug] Cold data (cooldown remote files) can be wrongly deleted after full clone, causing data loss

# [Bug] Cold data (cooldown remote files) can be wrongly deleted after full clone, causing data loss

> 中文摘要见文末 / Chinese summary at the bottom.

## Summary

On tables using **storage policy / cold data tiering** (cooldown to remote object storage such as S3/COS/HDFS) with **replication num >= 2**, cold data files on remote storage can be **permanently deleted while still being referenced** by the current (latest) cooldown metadata, after a **full clone** triggered by cluster topology changes (scale in/out, spec change, replica balance, BE restart, bad-disk replica repair).

The result is a **dangling reference**: the tablet metadata still records the rowset as valid, but the corresponding `.dat` object is gone from remote storage. Queries then fail at `initialize storage reader` with `[NOT_FOUND] failed to get file size ... on remote`.

This is a **latent, silent data-loss bug**. It is not specific to any single deployment.

## Affected versions / branches

Verified by source inspection on the latest of each branch:

| Branch | BE defect (reuse clone-source `cooldown_meta_id`) | FE defect (no null guard on `cooldown_meta_id`) | Path | Affected |
|--------|:--:|:--:|------|:--:|
| branch-2.1 | YES | YES | `be/src/olap/task/engine_clone_task.cpp` | **YES** |
| branch-3.1 | YES | YES | `be/src/olap/task/engine_clone_task.cpp` | **YES** |
| branch-4.1 | YES | YES | `be/src/storage/task/engine_clone_task.cpp` (relocated) | **YES** |
| master | YES | YES | `be/src/storage/task/engine_clone_task.cpp` (relocated) | **YES** |

Note: 4.1/master only relocated the code from `be/src/olap/` to `be/src/storage/`; the defective logic is unchanged.

## Root cause

There are two cooperating defects:

### Defect ① (BE) — reusing the clone-source `cooldown_meta_id` after full clone

In `EngineCloneTask::_finish_full_clone()`, after a full clone the code regenerates a new
`cooldown_meta_id` **only when this replica is the cooldown replica**; otherwise it **reuses the
`cooldown_meta_id` carried by the clone source**:

```cpp
// be/src/{olap,storage}/task/engine_clone_task.cpp :: _finish_full_clone()
if (tablet->cooldown_conf_unlocked().cooldown_replica_id == tablet->replica_id()) {
// this replica is cooldown replica: generate a fresh id (correct)
tablet->tablet_meta()->set_cooldown_meta_id(UniqueId::gen_uid());
} else {
// BUG: reuse clone source's cooldown_meta_id.
// This wrongly "aligns" replicas' cooldown_meta_id while their underlying
// cooldowned rowsets (rowset_id / remote files) may actually differ.
tablet->tablet_meta()->set_cooldown_meta_id(cloned_tablet_meta->cooldown_meta_id());
}
```

The original in-code comment already describes the exact loss scenario:

```
// Replica A is cooldown replica, cooldown_meta_id=2,
// Replica B: cooldown_replica=A, cooldown_meta_id=1
// Replica A: full clone Replica A, cooldown_meta_id=1, but remote cooldown_meta is still with cooldown_meta_id=2
// After tablet report, FE finds all replicas' cooldowned data is consistent
// Replica A: confirm_unused_remote_files, delete some cooldowned data of cooldown_meta_id=2
// Replica B: follow_cooldown_data, cooldown_meta_id=2, data lost
```

The `else` branch reintroduces precisely this hazard.

### Defect ② (FE) — no null guard in `confirmUnusedRemoteFiles`

In `FrontendServiceImpl.confirmUnusedRemoteFiles()`, deletion of remote files is allowed once all
replicas' `cooldown_meta_id` are considered "the same". But there is **no null check**:

```java
// fe/fe-core/.../service/FrontendServiceImpl.java :: confirmUnusedRemoteFiles()
for (Replica replica : replicas) {
...
// BUG: no null guard. A replica that just finished clone and has not reported
// its cooldown meta yet (getCooldownMetaId()==null), or a null request meta id,
// can still be mis-judged "consistent" and let deletion proceed.
if (!info.cooldown_meta_id.equals(replica.getCooldownMetaId())) {
LOG.info("cooldown meta id are not same, tablet={}", info.tablet_id);
return;
}
}
```

### How they combine into data loss

1. Topology change triggers a full clone; a non-cooldown replica reuses the clone source's `cooldown_meta_id` (Defect ①), so replicas' `cooldown_meta_id` get "aligned" even though their remote rowsets differ.
2. FE `confirmUnusedRemoteFiles` sees equal `cooldown_meta_id` (and no null guard, Defect ②) and treats all replicas as consistent → approves deletion.
3. BE background `remove_unused_remote_files` (default every 6h) reclaims by "old cooldown term meta" and deletes the remote `.dat` files referenced by that old term — **but those files are still referenced by the latest term's meta** → dangling reference → data loss.

## Impact

- Any Doris cluster with **cold data tiering enabled** + **replica num >= 2** + a **full clone event** (scale in/out, spec change, balance, BE restart, replica repair) is at risk.
- Deletion is **delayed ~3–6h** after the triggering op (aligned with `remove_unused_remote_files_interval_sec`, default 21600s).
- The defect can trigger on the **very first** cooldown-replica switch (cooldown_term = 0/1); it does not require many rounds of clone to accumulate.

## Reproduction (outline)

1. Create a table with a storage policy that cools down to remote storage; set `replication_num = 3`.
2. Load data and wait until data is cooled down to remote (RemoteDataSize > 0).
3. Trigger a full clone on a non-cooldown replica (e.g., drop a replica's local data / scale in-out / balance).
4. Wait for the next `remove_unused_remote_files` cycle.
5. Query the affected partition → `[NOT_FOUND] failed to get file size ... on remote`.

## Fix

### Fix ① (BE): always regenerate `cooldown_meta_id` after clone; never reuse the clone source

```cpp
// be/src/{olap,storage}/task/engine_clone_task.cpp :: _finish_full_clone()
{
std::shared_lock cooldown_conf_rlock(tablet->get_cooldown_conf_lock());
// MUST always generate a brand new cooldown_meta_id after clone, and NEVER reuse the
// cooldown_meta_id carried by the clone source. Regenerating forces the cooldown replica
// to re-write remote cooldown meta and followers to re-follow, making metadata re-converge
// to the real files, which eliminates the wrong "alignment".
tablet->tablet_meta()->set_cooldown_meta_id(UniqueId::gen_uid());
}
```

### Fix ② (FE): add null guard in `confirmUnusedRemoteFiles`

```java
// fe/fe-core/.../service/FrontendServiceImpl.java :: confirmUnusedRemoteFiles()

// The requesting cooldown replica MUST report a valid cooldown_meta_id.
// A null value means its cooldown meta state is not settled (e.g. just cloned, not reported);
// deleting now may remove files still referenced elsewhere.
if (info.cooldown_meta_id == null) {
LOG.info("request cooldown_meta_id is null, skip delete, tablet={}", info.tablet_id);
return;
}
for (Replica replica : replicas) {
...
// A replica whose cooldownMetaId is not yet reported (null) MUST NOT be treated as
// "consistent". Require every replica to carry a non-null cooldownMetaId equal to the
// one reported by the cooldown replica; otherwise deletion is unsafe.
if (replica.getCooldownMetaId() == null
|| !info.cooldown_meta_id.equals(replica.getCooldownMetaId())) {
LOG.info("cooldown meta id are not same, tablet={}", info.tablet_id);
return;
}
}
```

## Mitigation (before the fix is deployed)

- Freeze topology: no scale in/out, spec change, or replica reduction on clusters with cold data.
- FE: `disable_balance = true` to stop replica balancing.
- BE: increase `remove_unused_remote_files_interval_sec` (e.g. to 604800 = 7d) to slow down background reclamation.
- Back up / keep upstream copies of critical cold data.

## How to verify a cluster (read-only, no object storage access needed)

- Via FE read-only SQL: `SHOW TABLETS FROM db.table` exposes `CooldownReplicaId` / `CooldownMetaId` per replica.
- **HIGH risk signal**: for a cold tablet, replicas' `CooldownMetaId` differ, or the cooldown replica's `CooldownMetaId` is empty.
- Deep verify (needs storage access): parse each tablet's cooldown `.meta` (`TabletMetaPB`), take `rs_metas[*].rowset_id_v2`, and check whether the corresponding remote `.dat` objects still exist. A referenced-but-missing object is confirmed loss.

---

## 中文摘要

**问题**：启用冷热分层（降冷到对象存储）且副本数 ≥ 2 的表，在集群扩缩容/变配/副本均衡/BE 重启等触发**全量 clone** 后，远端冷数据文件可能被**误删**，而当前最新的 cooldown 元数据仍在引用它 —— 形成**悬空引用**，查询报 `[NOT_FOUND] ... on remote`。这是一个**静默的数据丢失缺陷**。

**受影响分支**：社区 `branch-2.1 / branch-3.1 / branch-4.1 / master` **全部存在且未修复**（4.1/master 仅将代码从 `be/src/olap/` 重构到 `be/src/storage/`，缺陷逻辑不变）。

**根因（两处协同）**：
- **BE**：`_finish_full_clone()` 里非 cooldown replica **复用了 clone 源的 `cooldown_meta_id`**，导致各副本 meta_id 被错误“对齐”，但底层远端 rowset 实际不同。
- **FE**：`confirmUnusedRemoteFiles()` 仅凭 `cooldown_meta_id` 相等就放行删除，且**无空值保护**。
- 后台 `remove_unused_remote_files`（默认 6h）按旧 term 回收，把仍被最新 term 引用的文件一并删除。

**修复**：① BE clone 后**始终**重新生成 `cooldown_meta_id`，绝不复用 clone 源；② FE 增加 `cooldown_meta_id` 空值保护。

**触发特征**：删除滞后触发操作约 3–6h；**首次切主（term=0/1）即可触发**，无需多轮累积。

**缓解**：冻结拓扑变更；FE `disable_balance=true`；调大 BE `remove_unused_remote_files_interval_sec`；备份关键冷数据。

**自查（只读、无需访问对象存储）**：`SHOW TABLETS` 看各副本 `CooldownReplicaId`/`CooldownMetaId` —— 同一冷 tablet 副本间 meta_id 不一致、或 cooldown replica 的 meta_id 为空 = 高危。

Start with EngineCloneTask::_finish_full_clone() in be/src/olap/task/engine_clone_task.cpp or be/src/storage/task/engine_clone_task.cpp, depending on the branch, then inspect FrontendServiceImpl.confirmUnusedRemoteFiles(). Reproduce the full-clone and cooldown scenario described in the issue; done means clone metadata is regenerated safely and remote-file deletion is refused when cooldown metadata is missing or inconsistent.

Written by the indexing model from the issue text.