RocketChat / RocketChat/Rocket.Chat

High Cpu node 100%

Open
#25,968 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

from time to time our rocket stops working.
today analyzing better, we noticed an abnormal traffic
coming from the IP address 51.81.47.27.

when blocking this address and restarting the rocket everything is normal.
When we unlock this IP, the node returns to 100% cpu. when this happens the rocket stops working.

this IP address seems to be from the rocket cloud.
what's the logic?

de vez em quando nosso rocket para de funcionar.
hoje analisando melhor, percebemos um trafego anormal
vindo do endereco IP 51.81.47.27.

ao bloquear esse endereco e reiniciar o rocket tudo fica normal.
qunado desbloqueamos esse IP, o node volta a ficar com 100% de cpu. quando isso acontece o rocket para de funcionar.

esse endereco IP parece ser do rocket cloud.
qual a logica?

Steps to reproduce:
  1. if the rocket stops working, and the node is at 100%
  2. tcpdump -n -i eth0 net 51.81.0.0/16
  3. netstat -anp | grep 51.81.47.27 | cat -n
    (69 tcp 0 0 177.22.32.247:443 51.81.47.27:31706 ESTABELECIDA 3842/nginx: worker)
Expected behavior:

rocket stops working

Server Setup Information:
  • Version of Rocket.Chat Server: 3.18.3
  • Operating System: Ubuntu 16.04.4 LTS
  • Deployment Method: snap
  • Number of Running Instances: 1
  • DB Replicaset Oplog:
  • NodeJS Version: 12.22.1 - x64
  • MongoDB Version: 3.6.14

erro_rocket_1

Relevant logs:

when the node is at 100% cpu, no log is generated

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported 100% Node.js CPU state with tcpdump and netstat while checking the reported absence of logs. Use the Rocket.Chat 3.18.3, Ubuntu 16.04.4, snap, and single-instance setup; done means explaining the 51.81.47.27 traffic and preventing the server from becoming unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx, node.js, ubuntu
Domain
backend, devops, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.