RocketChat / RocketChat/Rocket.Chat

4.2.1 gives 'Illegal instruction' - SIGILL in node on some CPU types

Open
#23,938 14 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Dear all,

after the upgrade to 4.2.1 my Rocket.chat instance did not start again.

TO debug this I did some parallel installation with versions 4.2.0 and 4.2.1 with (cd /tmp/bundle/programs/server; npm i).
4.2.0 does start fine, 4.2.1stops after some seconds with 'Illegal instruction':

# MONGO_URL="mongodb://127.0.0.1:27017/rocketchat?replicaSet=001-rs" MONGO_OPLOG_URL="mongodb://127.0.0.1:27017/local?replicaSet=001-rs" ROOT_URL=https://HOSTNAME_REPLACED_HERE  DEPLOY_PLATFORM=ansible PORT=3000  /usr/local/n/versions/node/12.22.1/bin/node main.js
Illegal instruction

System is up-to-date Debian 10. Deployed with Ansible using https://github.com/RocketChat/Rocket.Chat.Ansible with some variables added - this worked fine for the past months. I set these variables inside Ansible:

vars:
      rocket_chat_include_nginx: false
      rocket_chat_pgp_command: gpg
      rocket_chat_tarball_gpg_key: 9DA31620334BD75D9DCB49F368818C72E52529D4
      rocket_chat_tarball_gpg_keyserver: hkp://keyserver.ubuntu.com:80
      #rocket_chat_tarball_gpg_keyserver: hkp://p80.pool.sks-keyservers.net:80
      # hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4

      rocket_chat_service_host: chat01-back.int.somesite.de
      rocket_chat_automatic_upgrades: true
      rocket_chat_node_version:  12.22.1
      rocket_chat_npm_version: 6.14.1

and

---
rocket_chat_mongodb_org_version: 4.2
rocket_chat_mongodb_gpg_key: E162F504A20CDF15827F718D4B7C549A058F8B6B
rocket_chat_mongodb_service_name: mongod
rocket_chat_mongodb_org_pkgs: true
rocket_chat_mongodb_packages:
  - mongodb-org
  - mongodb-org-server

rocket_chat_dist_specific_packages:
  - g++

Started with gdb:

# MONGO_URL="mongodb://127.0.0.1:27017/rocketchat?replicaSet=001-rs" MONGO_OPLOG_URL="mongodb://127.0.0.1:27017/local?replicaSet=001-rs" ROOT_URL=https://HOSTNAME_REPLACED_HERE  DEPLOY_PLATFORM=ansible PORT=3000  gdb /usr/local/n/versions/node/12.22.1/bin/node 
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/n/versions/node/12.22.1/bin/node...done.
(gdb) run main.js
Starting program: /usr/local/n/versions/node/12.22.1/bin/node main.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff7ab7700 (LWP 10497)]
[New Thread 0x7ffff72b6700 (LWP 10498)]
[New Thread 0x7ffff6ab5700 (LWP 10499)]
[New Thread 0x7ffff62b4700 (LWP 10500)]
[New Thread 0x7ffff5ab3700 (LWP 10501)]
[New Thread 0x7ffff52b2700 (LWP 10502)]
[Detaching after fork from child process 10503]
[New Thread 0x7ffff509e700 (LWP 10504)]
[Thread 0x7ffff509e700 (LWP 10504) exited]
[New Thread 0x7ffff509e700 (LWP 10505)]
[New Thread 0x7fffdeefe700 (LWP 10506)]
[New Thread 0x7fffde6fd700 (LWP 10507)]
[New Thread 0x7fffddefc700 (LWP 10508)]
[Detaching after fork from child process 10511]
[Detaching after fork from child process 10515]

Thread 1 "node" received signal SIGILL, Illegal instruction.
0x00007fffd6038dfb in ?? () from /srv/4.2.1/bundle/programs/server/npm/node_modules/sharp/build/Release/../.././vendor/8.11.3/linux-x64/lib/libvips-cpp.so.42
(gdb) 

This is it for the moment. Please ask about anything else which might help tracking this down. Or about what I might try myself here.

Best regards
Florian

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure by comparing the 4.2.0 and 4.2.1 bundles with the supplied Node.js command and main.js. Use the reported GDB location in programs/server/npm/node_modules/sharp/.../libvips-cpp.so.42 as the starting point, then investigate the CPU-specific SIGILL. Done means the affected installation starts without the illegal-instruction failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.