RocketChat / RocketChat/Rocket.Chat

Instance service starting twice

Open
#36,268 11 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

planned
Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

We have recently migrated our k8s stack to a newer hardware. The old one was using ubuntu 22.04 while the new one is running on Rocky Linux 8.
Once we do a rolling restart to make our rocketchat server to scale to the new hardware, it hits Error: listen EADDRINUSE: address already in use :::36125 error.

I have tried to use another VM to re-install a Rocky Linux to join to our k8s cluster and RocketChat service can start without the issue above.

I also try to turn the ubuntu server that running on our old hardware to Rocky Linux server with identical settings and RocketChat can be start without any issues.

The new hardwares we are running as below:
2xIntel(R) Xeon(R) Gold 6338N CPU (32Core64Threads)
2x64G Memory

While the old hardware are running as below:
2xIntel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz (12Core24Threads)
8x32G Memory

As we have 4 unit of server for each setup, I can confirm that the issue only happen on our new hardware. Logs as below.


Steps to reproduce:
  1. Get a newer hardware
  2. Try to run RocketChat newer thant 7.3.0 container in newer hardware
Expected behavior:

Server can be start normally

Actual behavior:

instance service was tried to run twice.
I not sure is due to race condition or any other issue. But I am not good in nodejs, hence I have no choice to seek for help.

Server Setup Information:
  • Version of Rocket.Chat Server: 7.7.1 (it happen after 7.3.0)
  • License Type: Community
  • Number of Users: 200
  • Operating System: Rocky Linux 8
  • Deployment Method: Kubernetes
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 22.16.0 - x64
  • MongoDB Version: 6.0.10
Client Setup Information
  • Desktop App or Browser Version:
  • Operating System:
Additional context
Relevant logs:

Server running on newer hardware hit EADDRINUSE

+----------------------------------------------+
|                SERVER RUNNING                |
+----------------------------------------------+
|                                              |
|  Rocket.Chat Version: 7.7.1                  |
|       NodeJS Version: 22.16.0 - x64          |
|      MongoDB Version: 6.0.10                 |
|       MongoDB Engine: wiredTiger             |
|             Platform: linux                  |
|         Process Port: 3000                   |
|             Site URL: http://127.0.0.1:3000  |
|     ReplicaSet OpLog: Enabled                |
|          Commit Hash: c1bf9b1296             |
|        Commit Branch: HEAD                   |
|                                              |
+----------------------------------------------+
Starting service: ee-settings
Starting service: settings
Starting service: ldap-enterprise
Starting service: license
Starting service: message-reads
Starting service: omnichannel-ee
Starting service: voip-freeswitch
Starting service: instance
[2025-06-21T06:54:52.647Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSIT: Connecting to the transporter...
[2025-06-21T06:54:52.648Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSPORTER: TCP server is listening on port 36125
[2025-06-21T06:54:52.648Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSPORTER: UDP Discovery is disabled.
[2025-06-21T06:54:52.648Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSPORTER: TCP Transporter started.
[2025-06-21T06:54:53.154Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/REGISTRY: '$node' service is registered.
[2025-06-21T06:54:53.154Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/REGISTRY: 'matrix' service is registered.
[2025-06-21T06:54:53.155Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/$NODE: Service '$node' started.
[2025-06-21T06:54:53.155Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/MATRIX: Service 'matrix' started.
[2025-06-21T06:54:53.155Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/BROKER: ✔ ServiceBroker with 2 service(s) started successfully in 508ms.
Starting service: ee-settings
Starting service: instance
[2025-06-21T06:54:54.700Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSIT: Connecting to the transporter...
[2025-06-21T06:54:54.700Z] ERROR 03f00c21-09ed-4f33-a96e-851050434318/TRANSPORTER: Server error. Error: listen EADDRINUSE: address already in use :::36125
    at Server.setupListenHandle [as _listen2] (node:net:1939:16)
    at listenInCluster (node:net:1996:12)
    at Server.listen (node:net:2101:7)
    at /app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp/tcp-reader.js:62:11
    at new Promise (<anonymous>)
    at TcpReader.listen (/app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp/tcp-reader.js:46:10)
    at TcpTransporter.startTcpServer (/app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp.js:155:22)
    at /app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp.js:122:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at listOnTimeout (node:internal/timers:549:9)
    at processTimers (node:internal/timers:523:7) {
  code: 'EADDRINUSE',
  errno: -98,
  syscall: 'listen',
  address: '::',
  port: 36125
}
[2025-06-21T06:54:54.702Z] WARN  03f00c21-09ed-4f33-a96e-851050434318/TRANSIT: Connection is failed. listen EADDRINUSE: address already in use :::36125
Starting service: apps-engine
Starting service: analytics
Starting service: authorization-livechat
Starting service: banner
Starting service: calendar
Starting service: ldap
Starting service: media
Starting service: meteor
Starting service: nps
Starting service: room
Starting service: sau-monitor
Starting service: voip-asterisk
Starting service: omnichannel
Starting service: omnichannel-voip
Starting service: team
Starting service: uikit-core-app
Starting service: push
Starting service: device-management
Starting service: video-conference
Starting service: upload
Starting service: message
Starting service: translation
Starting service: omnichannel-integration
Starting service: import
Starting service: omnichannel-analytics
Starting service: user
Starting service: presence
Starting service: presence
Starting service: authorization
Starting service: queue-worker
Starting service: omnichannel-transcript
Starting service: federation
Starting service: authorization
Starting service: queue-worker
Starting service: omnichannel-transcript
Starting service: federation
[2025-06-21T06:55:02.505Z] INFO  03f00c21-09ed-4f33-a96e-851050434318/TRANSIT: Reconnecting...
[2025-06-21T06:55:02.505Z] ERROR 03f00c21-09ed-4f33-a96e-851050434318/TRANSPORTER: Server error. Error: listen EADDRINUSE: address already in use :::36125
    at Server.setupListenHandle [as _listen2] (node:net:1939:16)
    at listenInCluster (node:net:1996:12)
    at Server.listen (node:net:2101:7)
    at /app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp/tcp-reader.js:62:11
    at new Promise (<anonymous>)
    at TcpReader.listen (/app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp/tcp-reader.js:46:10)
    at TcpTransporter.startTcpServer (/app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp.js:155:22)
    at /app/bundle/programs/server/npm/node_modules/moleculer/src/transporters/tcp.js:122:21
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at runNextTicks (node:internal/process/task_queues:69:3)
    at processTimers (node:internal/timers:520:9) {
  code: 'EADDRINUSE',
  errno: -98,
  syscall: 'listen',
  address: '::',
  port: 36125
}

server running on old hardware does not hit any issue

+----------------------------------------------+
|                SERVER RUNNING                |
+----------------------------------------------+
|                                              |
|  Rocket.Chat Version: 7.7.1                  |
|       NodeJS Version: 22.16.0 - x64          |
|      MongoDB Version: 6.0.10                 |
|       MongoDB Engine: wiredTiger             |
|             Platform: linux                  |
|         Process Port: 3000                   |
|             Site URL: http://127.0.0.1:3000  |
|     ReplicaSet OpLog: Enabled                |
|          Commit Hash: c1bf9b1296             |
|        Commit Branch: HEAD                   |
|                                              |
+----------------------------------------------+
Starting service: ee-settings
Starting service: settings
Starting service: ldap-enterprise
Starting service: license
Starting service: message-reads
Starting service: omnichannel-ee
Starting service: voip-freeswitch
Starting service: instance
[2025-06-21T07:04:29.790Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/TRANSIT: Connecting to the transporter...
[2025-06-21T07:04:29.793Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/TRANSPORTER: TCP server is listening on port 36427
[2025-06-21T07:04:29.793Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/TRANSPORTER: UDP Discovery is disabled.
[2025-06-21T07:04:29.794Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/TRANSPORTER: TCP Transporter started.
[2025-06-21T07:04:30.314Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/REGISTRY: '$node' service is registered.
[2025-06-21T07:04:30.315Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/REGISTRY: 'matrix' service is registered.
[2025-06-21T07:04:30.315Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/$NODE: Service '$node' started.
[2025-06-21T07:04:30.315Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/MATRIX: Service 'matrix' started.
[2025-06-21T07:04:30.316Z] INFO  7de30b58-871b-4d01-84cc-c9b14028f253/BROKER: ✔ ServiceBroker with 2 service(s) started successfully in 527ms.
Starting service: apps-engine
Starting service: analytics
Starting service: authorization-livechat
Starting service: banner
Starting service: calendar
Starting service: ldap
Starting service: media
Starting service: meteor
Starting service: nps
Starting service: room
Starting service: sau-monitor
Starting service: voip-asterisk
Starting service: omnichannel
Starting service: omnichannel-voip
Starting service: team
Starting service: uikit-core-app
Starting service: push
Starting service: device-management
Starting service: video-conference
Starting service: upload
Starting service: message
Starting service: translation
Starting service: omnichannel-integration
Starting service: import
Starting service: omnichannel-analytics
Starting service: user
Starting service: presence
Starting service: authorization
Starting service: queue-worker
Starting service: omnichannel-transcript
Starting service: federation
Starting service: ee-settings

the log was generate from packages/core-services/src/LocalBroker.ts

for await (const service of Array.from(this.pendingServices)) {
        console.log("<service log here>") <<<<
	const serviceInstance = this.services.get(service);
	if (serviceInstance) {
		await this.startService(serviceInstance);
	}
}

I believe this issue is similar with #35590

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/core-services/src/LocalBroker.ts, especially the pendingServices loop and the logged startService calls. Compare the duplicate instance and other service entries in the failing Kubernetes startup logs, then reproduce the reported startup sequence on the newer hardware or an equivalent setup. Done means the instance service starts once without repeated EADDRINUSE errors on port 36125.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, node.js, typescript
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.