RocketChat / RocketChat/Rocket.Chat
Rocket.Chat stability issue when many users disconnected and connected again
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 46.1k
- Forks
- 13.9k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 130
Description
Description:
Hi,
From time to time we have observing really serious Rocket.Chat stability issues. Duo to network problem, about half of our 3000 connected users are disconnected and connected again to Rocket.Chat. This situation caused under 100% processors load on Rocket docker containers and unavailability (messages greyed out, client disconnects). Also high containers RAM and Garbage Collector usage (screens below). It was probably caused by presence broadcast storm, because when we switched on "Disable Presence Broadcast", after couple of minutes everything back to normal. Switch off it again caused above problems and unavailability.
Permanent switch on "Disable Presence Broadcast" is not an option because user presence is not correct.
Steps to reproduce:
- Server with about 3000 users online
- In example restart reverse proxy (HAproxy, Nginx...)
- Users are disconnected and connected again
- Rocket.Chat is unavailable due to high load
Expected behavior:
Disconnect and connect again a lot of users same time not cause Rocket unavailable
Actual behavior:
Rocket is unavailable
Server Setup Information:
- Version of Rocket.Chat Server: 3.9.7
- Operating System: Centos7
- Deployment Method: docker-compose
- Number of Running Instances: 33
- DB Replicaset Oplog: YES
- NodeJS Version: v12.18.4
- MongoDB Version: 4.0.14
Client Setup Information
- Desktop App or Browser Version: All
- Operating System: All
Additional context
Docker contaniers CPU usage

Users sessions

Garbage collector

Heavy rocketchat.users collection updates

Relevant logs:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin with the presence broadcast behavior and the heavy rocketchat.users updates described in the report; reproduce the reconnect surge in the listed 3,000-user, 33-instance Docker Compose setup. Done means reconnecting users no longer makes Rocket.Chat unavailable while presence remains accurate; use the reported CPU, RAM, garbage collector, and users-collection symptoms to verify.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker-compose, mongodb, node.js, typescript
- Domain
- backend, databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100