RocketChat / RocketChat/Rocket.Chat

Multithreading problem when running on bare metal kubernetes

Open
#28,035 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

I have a rocketchat server running on a kubernetes cluster (v1.21.6), which runs on virtual machines. Recently, our infrastructure changed to a different kubernetes cluster (v1.23.5), which runs on bare-metal servers.
The server in the new cluster suffers from CPU throttling when CPU limits are set, even though the actual CPU use is much lower than the limits.
When researching the problem further, we noticed that in the old infrastructure server spawns around 10-30 threads, no matter what the load is, but in the new infrastructure, any action causes it to spawn 267-270 threads, which do not close afterwards, even when the server is idle.
This problem is not reproduced with other Node.js applications on this platform, so it seems to be an applicative, rather than ifrastructure problem.

Steps to reproduce:
Run rocketchat server 5.4.3 on bare metal kubernetes, and set any CPU limits.
Expected behavior:
Not opening many threads, and therefore no CPU throttling.

Actual behavior:
The performance is very slow, because of CPU throttling.

Server Setup Information:
Version of Rocket.Chat Server: 5.4.3
Operating System: -
Deployment Method: Kubernetes 1.23.5
Number of Running Instances: 20

Relevant logs:
There are no relevant logs (plus, the application runs offline, so attaching the logs would be quite hard).

Thanks

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce Rocket.Chat 5.4.3 on bare-metal Kubernetes 1.23.5 with CPU limits and compare its thread count with the older virtual-machine deployment. Start by investigating the deployment and runtime conditions described in the report; done means identifying why actions leave roughly 267–270 threads and confirming that the excessive threading and CPU throttling are resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, node.js
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.