apache / apache/rocketmq

IOPS was too high when mq was cleanning up expired files automatically

Open
#10,007 2 comments 0 reactions 0 assignees View on GitHub
module/doc
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
2d 20h
Merged PRs (30d)
26

Description

### Search before creation

- [x] I had searched in the [issues](https://github.com/apache/rocketmq/issues) and found no similar issues.

### Documentation Related

Cluster's flush type is SYNC;
Cluster's deployment type is 2m-2s;
Cluster's version is 4.9.7;
300-500 topics in cluster;
And other runtime settings are almost default.
I found the IOPS was too high when deleted expired files automatically in CommitLog, IndexFile and ConsumeQueue.
What reasons could lead to this? How to decrease IO.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the expired-file cleanup paths for CommitLog, IndexFile, and ConsumeQueue, then reproduce the behavior with the reported 4.9.7 cluster settings while measuring IOPS. Determine which cleanup operations cause the excess I/O and document a verified way to reduce it or a focused change, with results from the reproduction as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.