"Error: failed [502] Bad Gateway" then Push stops working

Description:

I am getting "Error: failed [502] Bad Gateway" in syslog, following which Push notifications stop working until i restart the server. This has happened three times in the last 2 weeks at different times.

Steps to reproduce:

Unknown

Expected behavior:

Push works

Actual behavior:

Push stops working

Server Setup Information:

Version of Rocket.Chat Server: 3.2.2
Operating System: Ubuntu 18.04.5 LTS (Bionic Beaver)
Deployment Method: snap
Number of Running Instances: 1
DB Replicaset Oplog: Enabled
NodeJS Version: 12.16.1 - arm64
MongoDB Version: 3.6.17

Client Setup Information

Affects both Android and iOS clients

Additional context

None

Relevant logs:

Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: Error: failed [502] Bad Gateway
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Object.exports.makeErrorByStatus (packages/http.js:176:10)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request._callback (packages/http.js:140:24)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.self.callback (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/node_modu
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.emit (events.js:311:20)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request. (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/node_module
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.emit (events.js:311:20)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage. (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/nod
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Object.onceWrapper (events.js:417:28)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage.emit (events.js:323:22)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at endReadableNT (_stream_readable.js:1204:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at processTicksAndRejections (internal/process/task_queues.js:84:21) {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: response: {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: statusCode: 502,
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: content: 'Bad Gateway',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: headers: {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: date: 'Sun, 13 Sep 2020 05:12:04 GMT',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: 'content-length': '11',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: 'content-type': 'text/plain; charset=utf-8',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: connection: 'close'
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: },
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: data: null
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: }
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: }

Start with the packages/http.js locations in the supplied syslog stack trace and investigate the 502 response that precedes push notifications stopping. Confirm the failure can be reproduced or isolated, then verify that push continues working without restarting the server.