Manual installed RC-Server crashes in the middle of the night

Description:

This morning our RC-Server was down. The RC-Service crashed in the middle of the night. Simply restarting the service brought it back to life and until now it is working fine again. I hope this was a one time crash, but I wanted to report it anyways. Maybe it helps. I also have trouble finding more detailed information logs other than '/var/logs/messages'. Any suggestion where to look else? RCs own logging in Administrator pane isn't helping since it was gone.

Steps to reproduce:

None

Expected behavior:

Not crash in the middle of the night with almost none user interaction.

Actual behavior:

Server Setup Information:

• Version of Rocket.Chat Server: 3.16.3
• Operating System: CentOS 7
• Deployment Method: Manual
• Number of Running Instances: 1
• DB Replicaset Oplog: Enabled
• NodeJS Version: 12.22.1
• MongoDB Version: 4.0.25

Client Setup Information

• Desktop App or Browser Version: 3.2.2
• Operating System: macOS Big Sur 11.4

Additional context

Relevant logs:

From 'systemctl status rocketchat':

rocketchat.service - The Rocket.Chat server
Loaded: loaded (/usr/lib/systemd/system/rocketchat.service; enabled; vendor preset: disabled)
Active: failed (Result: core-dump) since Tue 2021-07-20 00:58:16 CEST; 7h ago
Process: 16278 ExecStart=/usr/local/bin/node /opt/Rocket.Chat/main.js (code=dumped, signal=ILL)
Main PID: 16278 (code=dumped, signal=ILL)

Jul 20 00:58:14 rocketchat[16278]: 5: 0x1ac43ac v8::internal::compiler::PipelineImpl::Crea...ode]
Jul 20 00:58:14 rocketchat[16278]: 6: 0x1ac4f8e v8::internal::compiler::PipelineCompilatio...ode]
Jul 20 00:58:14 rocketchat[16278]: 7: 0xc713ad v8::internal::OptimizedCompilationJob::Prep...ode]
Jul 20 00:58:14 rocketchat[16278]: 8: 0xc75e18 [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat[16278]: 9: 0xc76730 v8::internal::Compiler::GetOptimizedCodeFor...ode]
Jul 20 00:58:14 rocketchat[16278]: 10: 0x10587b5 v8::internal::Runtime_CompileForOnStackRe...ode]
Jul 20 00:58:14 rocketchat[16278]: 11: 0x140de99 [/usr/local/bin/node]
Jul 20 00:58:16 systemd[1]: rocketchat.service: main process exited, code=dumped, status=4/ILL
Jul 20 00:58:16 systemd[1]: Unit rocketchat.service entered failed state.
Jul 20 00:58:16 systemd[1]: rocketchat.service failed.

From '/var/logs/messages':

Jul 20 00:58:14 rocketchat: #
Jul 20 00:58:14 rocketchat: # Fatal error in , line 0
Jul 20 00:58:14 rocketchat: # Check failed: IsFeedbackVector().
Jul 20 00:58:14 rocketchat: #
Jul 20 00:58:14 rocketchat: #
Jul 20 00:58:14 rocketchat: #
Jul 20 00:58:14 rocketchat: #FailureMessage Object: 0x7f4b2ddec6c0
Jul 20 00:58:14 rocketchat: 1: 0xa88a61 [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 2: 0x1a36cd5 V8_Fatal(char const*, ...) [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 3: 0x1a8ba86 v8::internal::compiler::JSFunctionRef::feedback_vector() const [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 4: 0x1abbebe void v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::GraphBuilderPhase

() [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 5: 0x1ac43ac v8::internal::compiler::PipelineImpl::CreateGraph() [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 6: 0x1ac4f8e v8::internal::compiler::PipelineCompilationJob::PrepareJobImpl(v8::internal::Isolate*) [
/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 7: 0xc713ad v8::internal::OptimizedCompilationJob::PrepareJob(v8::internal::Isolate*) [/usr/local/bin
/node]
Jul 20 00:58:14 rocketchat: 8: 0xc75e18 [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 9: 0xc76730 v8::internal::Compiler::GetOptimizedCodeForOSR(v8::internal::Handle<v8::internal::JSFunct
ion>, v8::internal::BailoutId, v8::internal::JavaScriptFrame*) [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 10: 0x10587b5 v8::internal::Runtime_CompileForOnStackReplacement(int, unsigned long*, v8::internal::I
solate*) [/usr/local/bin/node]
Jul 20 00:58:14 rocketchat: 11: 0x140de99 [/usr/local/bin/node]
Jul 20 00:58:14 kernel: traps: node[16278] trap invalid opcode ip:1a3b6f9 sp:7f4b2ddec698 error:0 in node[400000+26b5000]
Jul 20 00:58:14 abrt-hook-ccpp: Process 16278 (node) of user 1001 killed by SIGILL - dumping core
Jul 20 00:58:16 systemd: rocketchat.service: main process exited, code=dumped, status=4/ILL
Jul 20 00:58:16 systemd: Unit rocketchat.service entered failed state.
Jul 20 00:58:16 systemd: rocketchat.service failed.
Jul 20 00:58:16 abrt-server: Executable '/usr/local/bin/node' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Jul 20 00:58:16 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2021-07-20-00:58:14-16278' exited with 1
Jul 20 00:58:16 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2021-07-20-00:58:14-16278'

Start with the reported systemctl status rocketchat output and /var/log/messages, focusing on the Node.js SIGILL, V8 fatal error, and core-dump details. Compare the Rocket.Chat 3.16.3, CentOS 7, Node.js 12.22.1, and MongoDB 4.0.25 setup against a reproducible failure; done means identifying the crash cause and documenting or verifying a fix.