RocketChat / RocketChat/Rocket.Chat

Rocket.Chat stability issue when many users disconnected and connected again

Open
#21,182 60 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

Hi,
From time to time we have observing really serious Rocket.Chat stability issues. Duo to network problem, about half of our 3000 connected users are disconnected and connected again to Rocket.Chat. This situation caused under 100% processors load on Rocket docker containers and unavailability (messages greyed out, client disconnects). Also high containers RAM and Garbage Collector usage (screens below). It was probably caused by presence broadcast storm, because when we switched on "Disable Presence Broadcast", after couple of minutes everything back to normal. Switch off it again caused above problems and unavailability.
Permanent switch on "Disable Presence Broadcast" is not an option because user presence is not correct.

Steps to reproduce:
  1. Server with about 3000 users online
  2. In example restart reverse proxy (HAproxy, Nginx...)
  3. Users are disconnected and connected again
  4. Rocket.Chat is unavailable due to high load
Expected behavior:

Disconnect and connect again a lot of users same time not cause Rocket unavailable

Actual behavior:

Rocket is unavailable

Server Setup Information:
  • Version of Rocket.Chat Server: 3.9.7
  • Operating System: Centos7
  • Deployment Method: docker-compose
  • Number of Running Instances: 33
  • DB Replicaset Oplog: YES
  • NodeJS Version: v12.18.4
  • MongoDB Version: 4.0.14
Client Setup Information
  • Desktop App or Browser Version: All
  • Operating System: All
Additional context

Docker contaniers CPU usage
image

Users sessions
image

Garbage collector
image

Heavy rocketchat.users collection updates
image

Relevant logs:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the presence broadcast behavior and the heavy rocketchat.users updates described in the report; reproduce the reconnect surge in the listed 3,000-user, 33-instance Docker Compose setup. Done means reconnecting users no longer makes Rocket.Chat unavailable while presence remains accurate; use the reported CPU, RAM, garbage collector, and users-collection symptoms to verify.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, mongodb, node.js, typescript
Domain
backend, databases, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.