RocketChat / RocketChat/Rocket.Chat
Multithreading problem when running on bare metal kubernetes
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 46.1k
- Forks
- 13.9k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 130
Description
Description:
I have a rocketchat server running on a kubernetes cluster (v1.21.6), which runs on virtual machines. Recently, our infrastructure changed to a different kubernetes cluster (v1.23.5), which runs on bare-metal servers.
The server in the new cluster suffers from CPU throttling when CPU limits are set, even though the actual CPU use is much lower than the limits.
When researching the problem further, we noticed that in the old infrastructure server spawns around 10-30 threads, no matter what the load is, but in the new infrastructure, any action causes it to spawn 267-270 threads, which do not close afterwards, even when the server is idle.
This problem is not reproduced with other Node.js applications on this platform, so it seems to be an applicative, rather than ifrastructure problem.
Steps to reproduce:
Run rocketchat server 5.4.3 on bare metal kubernetes, and set any CPU limits.
Expected behavior:
Not opening many threads, and therefore no CPU throttling.
Actual behavior:
The performance is very slow, because of CPU throttling.
Server Setup Information:
Version of Rocket.Chat Server: 5.4.3
Operating System: -
Deployment Method: Kubernetes 1.23.5
Number of Running Instances: 20
Relevant logs:
There are no relevant logs (plus, the application runs offline, so attaching the logs would be quite hard).
Thanks
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce Rocket.Chat 5.4.3 on bare-metal Kubernetes 1.23.5 with CPU limits and compare its thread count with the older virtual-machine deployment. Start by investigating the deployment and runtime conditions described in the report; done means identifying why actions leave roughly 267–270 threads and confirming that the excessive threading and CPU throttling are resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, node.js
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100