RocketChat / RocketChat/Rocket.Chat

Rocketchat unstable after mongoDB restart (docker-composer v3 depends_on issue?)

Open
#25,654 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

After a mongoDB restart (docker recreate), rocketchat becomes unstable.

Not sure if depends_on in docker-composer v2 forced a rocketchat restart.
In that case, how to handle that in newer docker-composer syntax versions?

Steps to reproduce:
  1. recreate the mongoDB container
  2. see some clients not being able to send all messages (feels random)
  3. restart rocketchat container
Expected behavior:

Rocketchat fully recovers after a mongoDB restart/recreate or docker forces a Rocketchat restart in such case.

Actual behavior:

Rocketchat becomes unstable, and stays that way till it's restarted (left it hours)

Server Setup Information:
  • Version of Rocket.Chat Server: 4.6.3
  • Operating System: official docker image
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: 1
  • NodeJS Version: 14.18.3
  • MongoDB Version: 5.0.8
Additional context

Reading online this issue might not be there with the older docker-composer v2 "depends_on" syntax behavior.
I don't specify any version on top of my docker-composer file now.

Relevant logs:
Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
Exception in setInterval callback: MongoServerSelectionError: getaddrinfo ENOTFOUND mongo
     at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/sdam/topology.js:437:30)
     at listOnTimeout (internal/timers.js:557:17)
     at processTimers (internal/timers.js:500:7) {
   reason: TopologyDescription {
     type: 'Single',
     setName: null,
     maxSetVersion: null,
     maxElectionId: null,
     servers: Map(1) { 'mongo:27017' => [ServerDescription] },
     stale: false,
     compatible: true,
     compatibilityError: null,
     logicalSessionTimeoutMinutes: null,
     heartbeatFrequencyMS: 10000,
     localThresholdMS: 15,
     commonWireVersion: 13
   }
 }
 
 {"level":50,"time":"2022-05-26T14:46:12.417Z","pid":10,"hostname":"3d65f933ba3f","name":"LivechatEnterprise","section":"Queue","msg":"Error processing queue public","err":{"type":"MongoServerSelectionError","message":"getaddrinfo ENOTFOUND mongo","stack":"MongoServerSelectionError: getaddrinfo ENOTFOUND mongo\n    at Timeout._onTimeout (/app/bundle/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/sdam/topology.js:437:30)\n    at listOnTimeout (internal/timers.js:557:17)\n    at processTimers (internal/timers.js:500:7)\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at app/models/server/raw/OmnichannelQueue.ts:43:15\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40\n => awaited here:\n    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)\n    at ee/app/livechat-enterprise/server/lib/LivechatEnterprise.js:269:4\n    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40","name":"MongoServerSelectionError","reason":{"type":"Single","setName":null,"maxSetVersion":null,"maxElectionId":null,"servers":{},"stale":false,"compatible":true,"compatibilityError":null,"logicalSessionTimeoutMinutes":null,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"commonWireVersion":13}},"msg":"getaddrinfo ENOTFOUND mongo"}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the MongoDB container recreation with the Docker Compose deployment configuration and compare the relevant depends_on behavior. Inspect the failures referenced in app/models/server/raw/OmnichannelQueue.ts:43 and app/livechat-enterprise/server/lib/LivechatEnterprise.js:269. Done means Rocket.Chat either recovers after MongoDB restarts or the required restart behavior is clearly established and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, mongodb, node.js
Domain
databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.