[Bug] (storage policy) Cooled rowset reads fail with [E-206]get fs failed after BE restart in non-cloud mode: storage resource map not re-synced, self-heal gated by is_cloud_mode
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 531
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
- Apache Doris **4.0.3** (community release, tag `4.0.3-rc03`), **non-cloud mode** (1 FE + 1 BE on one node)
- Remote storage: Alibaba Cloud OSS (S3-compatible), registered via `CREATE S3 RESOURCE` + `CREATE STORAGE POLICY` with `cooldown_ttl = 600s`
- Affected table: AUTO PARTITION duplicate-key table; ETL runs `INSERT OVERWRITE` on its partitions every minute
- The relevant code is unchanged in 4.0.4–4.0.8 / branch-4.0 (verified at source level)
### What's Wrong?
## Summary
In **non-cloud mode**, after a **BE restart**, queries that need to open segments of rowsets already cooled down to S3/OSS fail and keep failing with `[E-206] get fs failed`: the BE-side in-memory storage-resource map is empty after restart and is never re-populated, and the read-path self-heal is gated by `config::is_cloud_mode()`.
## Root cause chain (verified in 4.0.3-rc03 source)
1. BE keeps storage resources only in an in-memory global map (`s_storage_resource_mgr` in `be/src/olap/storage_policy.cpp`). It is not persisted.
2. The map is populated **only** by the `PUSH_STORAGE_POLICY` agent task (`push_storage_policy_callback` in `be/src/agent/task_worker_pool.cpp` → `update_s3_resource()` → `put_storage_resource()`).
3. After a BE restart the map is empty. FE does not re-push storage policies/resources to a restarting BE (the tablet report already carries the BE's known resource ids/versions, but no re-push is triggered).
4. When a query opens a cooled segment, `RowsetMeta::remote_storage_resource()` (`be/src/olap/rowset/rowset_meta.cpp:167`) misses the map; its self-heal (`sync_storage_vault()`) is **gated by `config::is_cloud_mode()`** (line 178), so in non-cloud mode it returns `InternalError("cannot find storage resource. resource_id={}")` (line 188) without any re-sync attempt.
5. `RowsetMeta::physical_fs()` (line 113) logs that error and returns `nullptr`; `BetaRowset::load_segment` (`be/src/olap/rowset/beta_rowset.cpp:190`) then fails the query with the bare `[E-206] get fs failed`.
6. Nothing heals it: the query keeps failing on every attempt until something else re-populates the map (a new cooldown triggering a policy push, or a partition drop/rebuild).
## Production evidence (observational — we did not run a controlled restart experiment)
Single-node 4.0.3 cluster, one S3-compatible (OSS) resource; minute-level `INSERT OVERWRITE` ETL on local (non-cooled) partitions kept working throughout everything below.
- 2026-09-04 15:37 BE restart (upgrade; our be.WARNING window opens 09-07 12:12, already inside the resulting error storm): queries against partitions cooled to OSS failing with `[E-206] get fs failed` (~92K warnings), which only stopped when those partitions were dropped/recreated (09-07 15:55).
- 09-07 15:50 the policy was attached to two other partitions and they cooled down successfully — a successful cooldown needs the resource on the BE, so a push must have re-populated the resource map between the restart and that cooldown.
- 2026-09-10 10:25 BE restart. The first warning after BE came back (10:25:46.927) is `cannot find storage resource. resource_id=...` (twice), then >150K `[E-206]` warnings within ~5 hours from 6,720 distinct queries, until the BE process was stopped again at 15:22. Between the two storms (multi-day window) there is not a single such warning.
## Observability traps
- `SELECT count(*)` on the affected table succeeds (it only reads rowset meta), so health checks look fine.
- Point / `LIMIT` queries that stop early can succeed via the segment cache, so the failure looks intermittent.
- Deterministic verification needs a real full scan of the cooled tablet (e.g. `ORDER BY` on a non-key column).
### What You Expected?
Cooled (remote) rowsets should remain readable after a BE restart in non-cloud mode. Concretely, any of:
1. FE re-pushes storage policies/resources to a BE when it registers/reports after a restart (the report already carries resource ids/versions);
2. or the read path self-heals on a map miss in non-cloud mode too — remove the `config::is_cloud_mode()` gate in `RowsetMeta::remote_storage_resource()` and re-fetch the resource from FE;
3. or BE persists / rebuilds the resource map at startup.
Independently, the failure should be actionable instead of `[E-206] get fs failed`: `BetaRowset::load_segment` should include `resource_id` in the message (nearby code already does this, e.g. beta_rowset.cpp:100/157), and the underlying `cannot find storage resource` root cause should not stay hidden.
### How to Reproduce?
**We have NOT run a controlled reproduction.** The steps below are derived from the 4.0.3 source analysis above and are consistent with what we observed in production, but they are untested — treat them as a suggested path; corrections welcome.
1. Non-cloud cluster, 1 FE + 1 BE.
2. `CREATE S3 RESOURCE` (any S3-compatible storage), then `CREATE STORAGE POLICY ... PROPERTIES("cooldown_ttl"="60s")`.
3. Create a table with the policy attached, load data once, wait for rowsets to cool down to S3; verify a full scan works (e.g. `SELECT ... ORDER BY `).
4. Restart the BE, and do NOT load new data before step 5 — anything that makes FE push the policy again re-populates the BE map and closes the failure window.
5. Run the same full-scan query; per the code it should fail with `[E-206] get fs failed`, and be.WARNING should log `cannot find storage resource. resource_id=...` on every attempt.
Why reproduction may look non-deterministic (matches our production experience):
- Only queries that actually open cooled segments fail: `SELECT count(*)` reads meta only and always succeeds; `LIMIT`-style queries can succeed via early termination + segment cache.
- The failure window is "from BE restart until the next policy re-push"; after a re-push every query succeeds again.
### Anything Else?
- Related open issue: #63669 hits the same `get fs failed` family on a different path (clone). Our case is the plain read path after restart in non-cloud mode.
- Environment details: single node, S3-compatible object storage; storage policy attached to AUTO PARTITION table partitions; heavy minute-level `INSERT OVERWRITE` workload (writes unaffected).
- We also observed a separate `[E-3115] version not continuous for mow` problem on a mow table the same day; I can file it separately once I have a minimal repro.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by reading RowsetMeta::remote_storage_resource() and physical_fs() in be/src/olap/rowset/rowset_meta.cpp, then trace storage-resource population through push_storage_policy_callback(), update_s3_resource(), and put_storage_resource(). Use the suggested BE-restart and full-scan scenario to validate the failure window. Done means cooled rowsets remain readable after restart in non-cloud mode and failures expose the storage resource context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100