apache / apache/bifromq

dist_data文件泄漏

Open
#225 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
792
Forks
95
Avg merge
10h 45m
Merged PRs (30d)
4

Description

- 使用的 BifroMQ 版本:3.3.5
- 服务器规格配置:
mqtt01-05共5个节点,其中
mqtt01:16C32G,担任主节点
mqtt02-05:8C16G,担任子节点

现象
上线运行后,mqtt01的内存和磁盘呈持续上涨趋势,持续运行一个月后磁盘占用增加了28G,内存使用增加了3G,mqtt02-04无此问题,无法自动恢复,需要人为重启mqtt-01,重启之后内存和磁盘能自动清理掉,但随后仍以相同的趋势不断上涨
mqtt01磁盘趋势
Image

mqtt-01内存趋势

Image

mqtt-02-05磁盘趋势
Image

mqtt-02-05内存趋势

Image

分析
dist server的rocksdb在进行compaction时存在文件泄漏问题,导致sst文件没有合理压缩释放,文件持续增长
mqtt-01sst文件数

Image

mqtt02-05sst文件数
Image

mqtt-01 、02rocksdb日志

[mqtt01-02rocksdb日志.zip](https://github.com/user-attachments/files/25039358/mqtt01-02rocksdb.zip)

辅助信息
当前实时连接数
mqtt-01连接数只有51,主要用于共享订阅客户端处理消息,不对外设备开放连接,mqtt02-05单台机器有1.3w连接数,总5.6w连接

Image

新建连接数和断开连接数(每分钟)
mqtt-02-05每分钟单台新建连接数为6000,断开连接数和新建连接数解决,也为6千,每分钟为2.4w的新建连接和新建断连

Image

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the attached RocksDB logs and compare SST-file growth and compaction behavior on mqtt01 versus mqtt02-05. Reproduce the issue under the reported connection and disconnect churn, then trace the dist server's RocksDB compaction path. Done means SST files, disk usage, and memory no longer grow continuously on the primary node.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.