RocketChat / RocketChat/Rocket.Chat

Metrics of last restarted instance looks a bit overloaded

Open
#22,254 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

We have docker deployment with 25 instances for users connections and 2 instances for present monitor without users (actually one with presence monitor enabled).
But we have couple instances with strange load according to Rocket chat Metrics.
That instances have completely similar configurations in docker-compose.yml file and normal users amount (not more then other instances)

Steps to reproduce:
  1. Setup docker deployment and Metrics monitoring
  2. Check Metrics for some graphs
  3. Few instances behave strange
Expected behavior:

All instances should have similar load and metrics.

Actual behavior:

Here is a metric size graph
image
As you can see 5 instance on 2 server have much more metrics size
image
And another screen right now
image
And 4 instance on that same 2 server too
image
Event loop lag is bigger then other instances
image
image
Pod heap same as others
image
But amount of WS sessions is even less then other instances
image
image

Server Setup Information:
  • Version of Rocket.Chat Server: 3.15.0
  • Operating System: CentOS7
  • Deployment Method: docker
  • Number of Running Instances: 25
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 12.22.1
  • MongoDB Version: 4.2.14 WiredTiger
Client Setup Information
  • Desktop App or Browser Version: 3.2.2
  • Operating System: Windows 10
Additional context

That graphs collected at weekend with really low load, but metrics for that few instances behave same all times.
In general all 5 (last) instances have more metrics size and event loop lag.

We restart every instances one-by-one and in each server I restart from 5 to 1 instances (5-4-3-2-1)
So 5 instance of 5 server restarted first and 1 instance of 2 server restarted last (actually 1 instance of 1 server restarted last, but 1 server holds 2 instances without user sessions)

Relevant logs:

No

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the docker-compose.yml configurations and the supplied metrics graphs, comparing the affected instances with the others and accounting for the restart order. Reproduce or inspect the metrics behavior under the stated Rocket.Chat, NodeJS, MongoDB, and Docker versions; done means identifying why specific instances show higher metric size and event-loop lag and documenting or validating the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, mongodb, node.js
Domain
backend, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.