matrixorigin / matrixorigin/matrixone
[Bug]: Backup fails with Error 20405 after restore: file restore-dev-.../data/... is not found
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
Related historical issues (similar `20405 file ... is not found`):
- #22242 After the clone database is GC, a bug occurs where the file “is not found”
- #19268 file ... is not found when executing queries on standby mo
- #10763 偶现 Error 20405 (HY000): file ... is not found
### Branch Name
main (mo-backup image: `main-20260213-fix`)
### Commit ID
unknown (cluster image tag: `registry.cn-hangzhou.aliyuncs.com/mocloud/mo-backup:main-20260213-fix`, digest `sha256:bb38ed10d2935fac0b68623c89087cf273415e7eea33eb8418bc363debf223b1`)
### Other Environment Information
```Markdown
- Cluster: freetier-01 (cn-dev)
- Namespace: freetier-01
- Host: freetier-01.cn-hangzhou.cluster.cn-dev.matrixone.tech:6001
- Backup Job/Pod: backup-260908-100944 / backup-260908-100944-7qg9x
- Backup tool: /mo_br backup
- Backup target: s3://moc-backup-dev/backup-260908-100944
- OSS endpoint: http://oss-cn-hangzhou-internal.aliyuncs.com (cn-hangzhou)
- User: dump?role=__MO_OB__
- Parallelism: 150
- Resources: cpu 2500m, memory request 4Gi / limit 10Gi
```
### Actual Behavior
`mo_br backup` connects to MO successfully and finishes Preparation (including Aliyun OSS client setup), then fails in **Backup From MO** with:
```text
Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
```
Pod exits with `exitCode: 1`, phase `Failed`.
Key timeline (UTC):
- 2026-09-08 10:09:45 Connect MO / Test Connection / Preparation: success
- 2026-09-08 10:10:14 Backup From MO: failed with 20405
Important detail:
- The backup output filepath is `backup-260908-100944/full-...`
- The missing file path is under an **old restore prefix**: `restore-dev-20260213235245/data/...`
This strongly suggests catalog/object metadata still references restore leftover object keys whose physical files are already gone (GC/cleanup/incomplete path rewrite after restore).
Relevant log excerpt:
```text
===== Connect MO =====
Done!
===== Test Connection =====
Done!
===== Preparation =====
... new object storage {"Bucket":"moc-backup-dev", ...}
Done!
===== Backup From MO =====
ERROR backup/backup.go:94 do backup failed: Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
Error: Error 20405 (HY000): file restore-dev-20260213235245/data/01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000 is not found
```
### Expected Behavior
After restore, subsequent `mo_br backup` should succeed:
1. Either all referenced data files remain readable, or
2. Catalog/object metadata no longer references deleted restore-prefix files.
Backup should not fail because of stale `restore-dev-...` object keys.
### Steps to Reproduce
```Markdown
1. Restore a cluster/data into an environment that uses (or previously used) a restore filepath prefix like `restore-dev-YYYYMMDDHHMMSS`.
2. Later run `mo_br backup` against the restored cluster, writing to a new S3 filepath (e.g. `backup-YYMMDD-HHMMSS`).
3. Observe backup fail in Backup From MO with:
Error 20405 (HY000): file restore-dev-.../data/_00000 is not found
```
Observed once on:
- Job: `backup-260908-100944`
- Pod: `backup-260908-100944-7qg9x`
- Namespace: `freetier-01`
### Additional information
Current analysis / suspected root cause:
1. Not auth/connectivity/OSS credential failure (those stages succeeded).
2. Failure happens while reading source files requested by MO metadata.
3. Missing object is under historical restore prefix `restore-dev-20260213235245`, not the current backup target path.
4. Likely metadata/object-lifecycle inconsistency after restore (stale object key retained after GC/cleanup, or restore did not fully rewrite object references).
Suggested investigation:
1. Check whether object `01a07efd-5c71-76e3-86ff-10e8e4f867e9_00000` still exists under `restore-dev-20260213235245/data/`.
2. Check why catalog/object meta still references that restore-prefix path after restore.
3. Correlate with GC / cleanup around the 2026-02-13 restore.
4. Confirm whether ordinary queries on the same cluster also hit intermittent 20405 for the same UUID.
Contributor guide
Assessment
This issue has not been assessed yet.