RocketChat / RocketChat/Rocket.Chat
Prometheus Metric Port can't established when running multiple instances on one machine
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 46.1k
- Forks
- 13.9k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 130
Description
Description:
When running multiple instances on one machine the Prometheus port can only be used once. Other node processes running on the same machine can't use the port for their metric interface when another process already listens on it.
Steps to reproduce:
- create machine and start multiple rocketchat node processes (like described in: https://docs.rocket.chat/installation/manual-installation/multiple-instances-to-improve-performance)
- Enable Prometheus Metrics (Settings - Logs - Prometheus)
Expected behavior:
Multiple processes and listeners are started.
Actual behavior:
Only one additional port is opened and some of the logs are showing that they can't open a port
Server Setup Information:
- Version of Rocket.Chat Server: 3.3.3
- Operating System: CentOS 7.6
- Deployment Method: ansible/tar
- Number of Running Instances: 14
- DB Replicaset Oplog: yes
- NodeJS Version: 12.18.1
- MongoDB Version: 3.6.18
Client Setup Information
- Desktop App or Browser Version: Browser - Firefox 77
- Operating System: Windows
Additional context
We also tried to set OVERWRITE_SETTING_Prometheus_Port for each systemd unit we are running to have each process assigning an individual port, but it seems the port is taken from the db
Relevant logs:
Jul 02 12:06:18 rocketchat-0.novalocal RocketChat[7206]: Exception in callback of async function: Error: listen EADDRINUSE: address already in use 0.0.0.0:9458
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue by running multiple Rocket.Chat Node processes with Prometheus enabled, using the documented multiple-instance setup and distinct systemd settings. Trace how the Prometheus port is read from settings versus each process environment, then verify that every instance can start its metrics listener without EADDRINUSE errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100