apache / apache/doris

[Bug] BE Disk IO Saturation with High Self-Heal Rate Causing S3 Cache Persistence Failure

Open
#64,657 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Doris 4.1.0-rc03 (cloud mode, storage-compute separated)

### What's Wrong?

A specific BE node exhibits the following symptoms:

1. **High disk IO utilization** — consistently above 90%, while other nodes are normal.
2. **`cached_remote_reader_self_heal_on_not_found`** counter is 19,388 — 5-20x higher than other nodes in the same cluster.
3. **Data fetched from S3 cannot be effectively cached** — each query triggers remote reads (~2.5s for 300KB), and `BytesWriteIntoCache` is 49-70MB per query, yet subsequent queries still miss cache and fall back to S3.

某个BE IO Util 很高 (90%以上),cached_remote_reader_self_heal_on_not_found 很高,S3查询后的数据无法有效被缓存
IO 这么高。应该不是不均衡导致,5个BE,查了tablet分布和compaction都是比较均匀的。

Image

同样的查询,其他节点能走磁盘缓存,这个节点就是走不了,查询根本没缓存进来。
然后导致这一台CPU很高,拖慢了查询速度

Image

### What You Expected?

I'd like help diagnosing the root cause of this issue. I'm willing to provide any logs, metrics, or configuration details needed. Hoping to get this resolved.
帮忙排查出问题的原因,我愿意提供我一切能提供的,希望能得到解决。翻译成英语

### How to Reproduce?

_No response_

### Anything Else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start with the BE disk IO metrics, the cached_remote_reader_self_heal_on_not_found counter, and S3 cache behavior described in the report; compare this node with the other BE nodes and gather the requested logs, metrics, and configuration details. Done means identifying the cause of the failed cache persistence and confirming that subsequent queries use the disk cache instead of repeatedly reading from S3.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
backend, cloud, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.