matrixorigin / matrixorigin/matrixone

[Bug]: Backup fails with Error 20405 after restore: file restore-dev-.../data/... is not found

Open
#28,473 3 comments 0 reactions 1 assignee Claimed by @LeftHandCold View on GitHub
kind/bug severity/s0
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same bug?

- [x] I have checked the existing issues.

Related historical issues (similar `20405 file ... is not found`):
- #22242 After the clone database is GC, a bug occurs where the file “is not found”
- #19268 file ... is not found when executing queries on standby mo
- #10763 偶现 Error 20405 (HY000): file ... is not found

### Branch Name

main (mo-backup image: `main-20260213-fix`)

### Commit ID

unknown (cluster image tag: `registry.cn-hangzhou.aliyuncs.com/mocloud/mo-backup:main-20260213-fix`, digest `sha256:bb38ed10d2935fac0b68623c89087cf273415e7eea33eb8418bc363debf223b1`)

### Other Environment Information

```Markdown
- Cluster: freetier-01 (cn-dev)
- Namespace: freetier-01
- Host: freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech:6001
- Backup Job/Pod: backup-260908-100944 / backup-260908-100944-7qg9x
- Backup tool: /mo_br backup
- Backup target: s3://moc-backup-dev/backup-260908-100944
- OSS endpoint: http://oss-cn-hangzhou-internal.aliyuncs.com (cn-hangzhou)
- User: dump?role=__MO_OB__
- Parallelism: 150
- Resources: cpu 2500m, memory request 4Gi / limit 10Gi
```

### Actual Behavior

`mo_br backup` connects to MO successfully and finishes Preparation (including Aliyun OSS client setup), then fails in **Backup From MO** with:

```text
Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
```

Pod exits with `exitCode: 1`, phase `Failed`.

Key timeline (UTC):
- 2026-09-08 10:09:45 Connect MO / Test Connection / Preparation: success
- 2026-09-08 10:10:14 Backup From MO: failed with 20405

Important detail:
- The backup output filepath is `backup-260908-100944/full-...`
- The missing file path is under an **old restore prefix**: `restore-dev-20260213235245/data/...`

This strongly suggests catalog/object metadata still references restore leftover object keys whose physical files are already gone (GC/cleanup/incomplete path rewrite after restore).

Relevant log excerpt:

```text
===== Connect MO =====
Done!
===== Test Connection =====
Done!
===== Preparation =====
... new object storage {"Bucket":"moc-backup-dev", ...}
Done!
===== Backup From MO =====
ERROR backup/backup.go:94 do backup failed: Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
Error: Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
```

### Expected Behavior

After restore, subsequent `mo_br backup` should succeed:
1. Either all referenced data files remain readable, or
2. Catalog/object metadata no longer references deleted restore-prefix files.

Backup should not fail because of stale `restore-dev-...` object keys.

### Steps to Reproduce

```Markdown
1. Restore a cluster/data into an environment that uses (or previously used) a restore filepath prefix like `restore-dev-YYYYMMDDHHMMSS`.
2. Later run `mo_br backup` against the restored cluster, writing to a new S3 filepath (e.g. `backup-YYMMDD-HHMMSS`).
3. Observe backup fail in Backup From MO with:
Error 20405 (HY000): file restore-dev-.../data/_00000 is not found
```

Observed once on:
- Job: `backup-260908-100944`
- Pod: `backup-260908-100944-7qg9x`
- Namespace: `freetier-01`

### Additional information

Current analysis / suspected root cause:
1. Not auth/connectivity/OSS credential failure (those stages succeeded).
2. Failure happens while reading source files requested by MO metadata.
3. Missing object is under historical restore prefix `restore-dev-20260213235245`, not the current backup target path.
4. Likely metadata/object-lifecycle inconsistency after restore (stale object key retained after GC/cleanup, or restore did not fully rewrite object references).

Suggested investigation:
1. Check whether object `01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000` still exists under `restore-dev-20260213235245/data/`.
2. Check why catalog/object meta still references that restore-prefix path after restore.
3. Correlate with GC / cleanup around the 2026-02-13 restore.
4. Confirm whether ordinary queries on the same cluster also hit intermittent 20405 for the same UUID.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.