Email and push notifications randomly stop sending - requires server restart each time to fix

Description:

Email and push notifications randomly stop sending for no apparent reason. The notifications are inserted into the rocketchat email_notifications_queue collection in the DB but are never processed. No errors in the logs. Once these notifications stop sending, the only way to get them to send again is to restart the server. Upon restart, everything that was sitting in email_notifications_queue is then sent all at once.

After looking at the code and adding some logging into the NotificationQueue.ts file, it appears that the worker() function randomly stops running. The server then needs to be restarted so that Notification.initWorker() runs again to kick off the worker() job. In my case, this happens anywhere from a few hours to a few days but at least happens once a week and on a very low volume instance of RocketChat.

I think the implementation of the worker() logic is quite fragile as it has no way to recover if something goes wrong. Perhaps it would make sense to use the littledata:synced-cron package for this since it's basically a scheduled job. Or at least use the synced-cron package to check say every 5 minutes to make sure the worker is still running and if not, call Notification.initWorker() to start it up again.

Steps to reproduce:

1. Send a DM to an offline user.
2. See if the offline user got the email and push notification.

Expected behavior:

Offline user got the notification(s)

Actual behavior:

Sometimes they get the notification(s), sometimes they don't

Server Setup Information:

• Version of Rocket.Chat Server: 3.7.4
• Operating System: Ubuntu 18.04
• Deployment Method: Manual (https://docs.rocket.chat/installation/manual-installation/ubuntu)
• Number of Running Instances: 1
• DB Replicaset Oplog: Enabled
• NodeJS Version: 12.14.0 - x64
• MongoDB Version: 4.2.11

Client Setup Information

• Desktop App or Browser Version: Chrome 87.0.4280.88 (Official Build) (x86_64)
• Operating System: Mac OS 10.15.7 Catalina

Additional context

Relevant logs:

Start by reading NotificationQueue.ts, especially worker() and Notification.initWorker(), then reproduce the failure by sending a DM to an offline user and checking whether notifications remain queued. Trace why the worker stops without logged errors. Done means queued email and push notifications resume without requiring a server restart.