RocketChat / RocketChat/Rocket.Chat

Prometheus Metric Port can't established when running multiple instances on one machine

Open
#18,125 8 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

When running multiple instances on one machine the Prometheus port can only be used once. Other node processes running on the same machine can't use the port for their metric interface when another process already listens on it.

Steps to reproduce:
  1. create machine and start multiple rocketchat node processes (like described in: https://docs.rocket.chat/installation/manual-installation/multiple-instances-to-improve-performance)
  2. Enable Prometheus Metrics (Settings - Logs - Prometheus)
Expected behavior:

Multiple processes and listeners are started.

Actual behavior:

Only one additional port is opened and some of the logs are showing that they can't open a port

Server Setup Information:
  • Version of Rocket.Chat Server: 3.3.3
  • Operating System: CentOS 7.6
  • Deployment Method: ansible/tar
  • Number of Running Instances: 14
  • DB Replicaset Oplog: yes
  • NodeJS Version: 12.18.1
  • MongoDB Version: 3.6.18
Client Setup Information
  • Desktop App or Browser Version: Browser - Firefox 77
  • Operating System: Windows
Additional context

We also tried to set OVERWRITE_SETTING_Prometheus_Port for each systemd unit we are running to have each process assigning an individual port, but it seems the port is taken from the db

Relevant logs:
Jul 02 12:06:18 rocketchat-0.novalocal RocketChat[7206]: Exception in callback of async function: Error: listen EADDRINUSE: address already in use 0.0.0.0:9458

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue by running multiple Rocket.Chat Node processes with Prometheus enabled, using the documented multiple-instance setup and distinct systemd settings. Trace how the Prometheus port is read from settings versus each process environment, then verify that every instance can start its metrics listener without EADDRINUSE errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.