RocketChat / RocketChat/Rocket.Chat

RocketChat Server crash - JavaScript heap out of memory - triggered by client

Open
#23,909 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

Rocketchat Server crashes / JavaScript heap out of memory

Steps to reproduce:

unknown -> not reproduceable in test (maybe not enough users active there)

  1. start electron rocketchat client 3.6.0
  2. connect to server
  3. right click on server -> clear cache
  4. right click on server -> clear storage cache
  5. client disconnects and server was in crashed state
Expected behavior:

no server crash

Actual behavior:

FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory

Server Setup Information:

Rocket.Chat Version: 4.1.2
NodeJS Version: 12.22.5 - x64
MongoDB Version: 4.4.10
Platform: linux
Process Port: 3000
Site URL: https://chat.redacted
ReplicaSet OpLog: Enabled
Commit Hash: 2aba8d2d82
Commit Branch: HEAD

  • Version of Rocket.Chat Server: 4.1.2
  • Operating System: Red Hat Enterprise Linux release 8.5 (Ootpa)
  • Deployment Method: custom
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: 12.22.5 - x64
  • MongoDB Version: 4.4.10
  • MongoDB Engine: wiredTiger
Client Setup Information
  • Desktop App or Browser Version: Rocketchat 3.6.0
  • Operating System: Windows10
Additional context
Relevant logs:
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 RangeError: Out of memory
    at Fiber.run (<anonymous>)
    at makeNewFiber (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:71:11)
    at FiberPool.run (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:92:37)
    at Function.Promise.asyncApply (/data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:135:22)
    at MessagesRaw.findOne (app/models/server/raw/BaseRaw.ts:110:15)
    at server/modules/watchers/watchers.module.ts:115:66
    at /data/rocketchat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 ---------------------------------
 Errors like this can cause oplog processing errors.
 Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
 Future node.js versions will automatically exit the process
 =================================
 === UnHandledPromiseRejection ===
 <--- Last few GCs --->
 [2067276:0x55e7c4b8c6a0] 42016041 ms: Scavenge 626.3 (653.5) -> 621.5 (659.0) MB, 81.7 / 0.9 ms  (average mu = 1.000, current mu = 0.882) allocation failure
 [2067276:0x55e7c4b8c6a0] 42016184 ms: Scavenge 631.6 (659.0) -> 626.8 (664.2) MB, 86.1 / 0.9 ms  (average mu = 1.000, current mu = 0.882) allocation failure
 [2067276:0x55e7c4b8c6a0] 42016432 ms: Scavenge 636.7 (664.2) -> 632.5 (664.5) MB, 189.5 / 0.9 ms  (average mu = 1.000, current mu = 0.882) allocation failure
 <--- JS stacktrace --->
 Cannot get stack trace in GC.
 FATAL ERROR: NewSpace::Rebalance Allocation failed - JavaScript heap out of memory
 1: 0x55e7c278e604 node::Abort() [/usr/bin/node]
 2: 0x55e7c267b6fa node::OnFatalError(char const*, char const*) [/usr/bin/node]
 3: 0x55e7c292651a v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
 4: 0x55e7c29267a6 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
 5: 0x55e7c2abb789  [/usr/bin/node]
 6: 0x55e7c2b0a0d8 v8::internal::MarkCompactCollector::Evacuate() [/usr/bin/node]
 7: 0x55e7c2b0d6b0 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/bin/node]
 8: 0x55e7c2ad049c v8::internal::Heap::MarkCompact() [/usr/bin/node]
 9: 0x55e7c2ad13cb v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/bin/node]
 10: 0x55e7c2ad1b57 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/bin/node]
 11: 0x55e7c2ad41b5 v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
 12: 0x55e7c2ad4729 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/bin/node]
 13: 0x55e7c2a9d932 v8::internal::Factory::New(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationType) [/usr/bin/node]
 14: 0x55e7c2aa6648 v8::internal::Factory::NewFunction(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::Context>, v8::internal::AllocationType) [/usr/bin/node]
 15: 0x55e7c2aa6ac5 v8::internal::Factory::NewFunctionFromSharedFunctionInfo(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::Context>, v8::internal::Handle<v8::internal::FeedbackCell>, v8::internal::AllocationType) [/usr/bin/node]
 16: 0x55e7c2dd5946 v8::internal::Runtime_NewClosure_Tenured(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
 17: 0x55e7c31081f9  [/usr/bin/node]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with app/models/server/raw/BaseRaw.ts and server/modules/watchers/watchers.module.ts, the locations named in the stack trace, and review how the Electron client cache-clearing steps affect the server. The reported reproduction is not reliable, so first establish a repeatable trigger and capture memory behavior. Done means the server no longer crashes with JavaScript heap exhaustion during this workflow and the failure has a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
electron, mongodb, nodejs, typescript
Domain
backend, databases, devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.