dist_data文件泄漏
- Dominant language
- Java
- Stars
- 792
- Forks
- 95
- Avg merge
- 10h 45m
- Merged PRs (30d)
- 4
Description
- 使用的 BifroMQ 版本:3.3.5
- 服务器规格配置:
mqtt01-05共5个节点,其中
mqtt01:16C32G,担任主节点
mqtt02-05:8C16G,担任子节点
现象
上线运行后,mqtt01的内存和磁盘呈持续上涨趋势,持续运行一个月后磁盘占用增加了28G,内存使用增加了3G,mqtt02-04无此问题,无法自动恢复,需要人为重启mqtt-01,重启之后内存和磁盘能自动清理掉,但随后仍以相同的趋势不断上涨
mqtt01磁盘趋势
mqtt-01内存趋势
mqtt-02-05磁盘趋势
mqtt-02-05内存趋势
分析
dist server的rocksdb在进行compaction时存在文件泄漏问题,导致sst文件没有合理压缩释放,文件持续增长
mqtt-01sst文件数
mqtt02-05sst文件数
mqtt-01 、02rocksdb日志
[mqtt01-02rocksdb日志.zip](https://github.com/user-attachments/files/25039358/mqtt01-02rocksdb.zip)
辅助信息
当前实时连接数
mqtt-01连接数只有51,主要用于共享订阅客户端处理消息,不对外设备开放连接,mqtt02-05单台机器有1.3w连接数,总5.6w连接
新建连接数和断开连接数(每分钟)
mqtt-02-05每分钟单台新建连接数为6000,断开连接数和新建连接数解决,也为6千,每分钟为2.4w的新建连接和新建断连
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the attached RocksDB logs and compare SST-file growth and compaction behavior on mqtt01 versus mqtt02-05. Reproduce the issue under the reported connection and disconnect churn, then trace the dist server's RocksDB compaction path. Done means SST files, disk usage, and memory no longer grow continuously on the primary node.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100