RocketChat Out of memory - v8 node OOMKilled even when setting --max-old-space-size

Description:

Since upgrading from 3.x to 4.5, we're constantly having the RocketChat process OOMKilled by v8. I've tried increasing the available heap size with --max-old-space-size to 12GB but the problem still occurs.

Happens everytime I try to launch a background LDAP sync during the day and randomly throughout day/night.

Steps to reproduce:

1. Go to Administration
2. Go to LDAP settings
3. Click on Sync Now

Expected behavior:

RocketChat server not crashing.

Actual behavior:

The process aborts.

Server Setup Information:

• Version of Rocket.Chat Server: 4.5.5
• Operating System: Ubuntu 18.04
• Deployment Method: tar
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version: v14.19.1
• MongoDB Version: 4.0.28 / mmapv1 (oplog Enabled)

Client Setup Information

• Desktop App or Browser Version: Happens from all cleints - web/mobile/desktop
• Operating System: N/A

Additional context

Here is how RocketChat is initialized in the systemd service:

ExecStart=/usr/bin/node --max-old-space-size=12288 --max_old_space_size=12288 /opt/Rocket.Chat/main.js

Relevant logs:

First it crashed with:

Apr 12 09:43:10 chat rocketchat[8786]: Exception in callback of async function: RangeError: Out of memory
Apr 12 09:43:10 chat rocketchat[8786]:     at Fiber.run (/opt/Rocket.Chat/programs/server/node_modules/fibers/fibers.js:92:16)
Apr 12 09:43:10 chat rocketchat[8786]:     at Object.handler (packages/simple_json-routes.js:104:8)
Apr 12 09:43:10 chat rocketchat[8786]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/simple_json-routes/node_modules/connect-route/lib/connect-route.js:106:12
Apr 12 09:43:10 chat rocketchat[8786]:     at runWithEnvironment (packages/meteor.js:1320:24)
Apr 12 09:43:10 chat rocketchat[8786]:     at packages/meteor.js:1333:14
Apr 12 09:43:10 chat rocketchat[8786]:     at /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
Apr 12 09:43:11 chat rocketchat[8786]:     at Fiber.run (<anonymous>)
Apr 12 09:43:11 chat rocketchat[8786]:     at Fiber.run (/opt/Rocket.Chat/programs/server/node_modules/fibers/fibers.js:92:16)
Apr 12 09:43:11 chat rocketchat[8786]:     at process.<anonymous> (packages/meteor.js:1334:31)
Apr 12 09:43:11 chat rocketchat[8786]:     at process.emit (events.js:412:35)
Apr 12 09:43:11 chat rocketchat[8786]:     at process.emit (domain.js:475:12)
Apr 12 09:43:11 chat rocketchat[8786]:     at process._fatalException (internal/process/execution.js:167:25)
Apr 12 09:43:12 chat systemd[1]: rocketchat.service: Main process exited, code=exited, status=7/NOTRUNNING

On the second try after the server restarted:

Apr 12 09:45:20 chat rocketchat[43074]: <--- Last few GCs --->
Apr 12 09:45:20 chat rocketchat[43074]: [43074:0x46d3cd0]   122674 ms: Scavenge 661.3 (694.5) -> 654.7 (698.7) MB, 76.7 / 1.5 ms  (average mu = 0.988, current mu = 0.952) allocation failure
Apr 12 09:45:20 chat rocketchat[43074]: [43074:0x46d3cd0]   122839 ms: Scavenge 666.1 (699.1) -> 659.7 (703.8) MB, 79.0 / 1.6 ms  (average mu = 0.988, current mu = 0.952) allocation failure
Apr 12 09:45:20 chat rocketchat[43074]: [43074:0x46d3cd0]   123006 ms: Scavenge 670.8 (703.8) -> 664.6 (709.1) MB, 80.6 / 1.5 ms  (average mu = 0.988, current mu = 0.952) allocation failure
Apr 12 09:45:20 chat rocketchat[43074]: <--- JS stacktrace --->
Apr 12 09:45:20 chat rocketchat[43074]: FATAL ERROR: Zone Allocation failed - process out of memory
Apr 12 09:45:20 chat rocketchat[43074]:  1: 0xa3aaf0 node::Abort() [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  2: 0x970199 node::FatalError(char const*, char const*) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  3: 0xbba45e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  4: 0xbba7d7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  5: 0x12606ae v8::internal::Zone::NewExpand(unsigned long) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  6: 0x1a955ed v8::internal::compiler::JSHeapBroker::JSHeapBroker(v8::internal::Isolate*, v8::internal::Zone*, bool, bool) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  7: 0x1ae9d71 v8::internal::compiler::PipelineData::PipelineData(v8::internal::compiler::ZoneStats*, v8::internal::Isolate*, v8::internal::OptimizedCompilationInfo*, v8::internal::compiler::PipelineStatistics*, bool) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  8: 0x1aedc9c v8::internal::compiler::PipelineCompilationJob::PipelineCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::BailoutId, v8::internal::JavaScriptFrame*) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]:  9: 0x1aedda8 v8::internal::compiler::Pipeline::NewCompilationJob(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, bool, v8::internal::BailoutId, v8::internal::JavaScriptFrame*) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]: 10: 0xc84f68  [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]: 11: 0xc86bdd v8::internal::Compiler::CompileOptimized(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::ConcurrencyMode) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]: 12: 0x10866b1 v8::internal::Runtime_CompileOptimized_Concurrent(int, unsigned long*, v8::internal::Isolate*) [/usr/bin/node]
Apr 12 09:45:20 chat rocketchat[43074]: 13: 0x1448e19  [/usr/bin/node]
Apr 12 09:45:20 chat systemd[1]: rocketchat.service: Main process exited, code=killed, status=6/ABRT

No repository file or test is identified. Start with the Administration > LDAP > Sync Now flow and the supplied Node/V8 out-of-memory logs, then trace the LDAP sync entry point. Done means the server no longer aborts during the sync and the failure is covered by an appropriate test.