Rocket.Chat auto reseting settings after a month of usage

Description:

Hi everyone, before of all I tried to search about my problem, but without success, so let's resume what is happening, I'm using a self hosted version of rocket chat since december 2020, now on january 2021, I'm facing issues with the data on mongodb, one day all of setting of rocketchat just reseted, all users gone, logo gone. I checked on mongodb database was there but in small size, I done a backup to recover data (not updated), And now it happened again

Steps to reproduce:

1. Don't know how to reproduce this, since I just installed rocket chat using the tutorial on the docs, tried to install on Digital Ocean and IBM Cloud

Expected behavior:

Setting never reset.

Actual behavior:

Settings being reseted for reason I dont know

Server Setup Information:

• Version of Rocket.Chat Server: 3.8.0
• Operating System: Ubuntu 18.04
• Deployment Method: tar I believe
• Number of Running Instances: 1
• DB Replicaset Oplog: local?
• NodeJS Version: 12.18.4
• MongoDB Version: 4.0.22

Client Setup Information

• Desktop App or Browser Version: Any
• Operating System: Any

Relevant logs:

I believe that logs will not help since I can't get the logs right after the issue happened.

I20210126-13:36:11.104(0) LocalStore: store created at  
I20210126-13:36:11.440(0) LocalStore: store created at  
I20210126-13:36:11.441(0) LocalStore: store created at  
I20210126-13:36:11.663(0) Setting default file store to GridFS 
I20210126-13:36:15.664(0) LocalStore: store created at  
I20210126-13:36:15.665(0) LocalStore: store created at  
I20210126-13:36:15.666(0) LocalStore: store created at  
I20210126-13:36:15.873(0) Setting default file store to GridFS 
I20210126-13:36:36.181(0) LocalStore: store created at  
I20210126-13:36:36.182(0) LocalStore: store created at  
I20210126-13:36:36.183(0) LocalStore: store created at  
I20210126-13:36:36.571(0) Updating process.env.MAIL_URL 
I20210126-13:36:36.732(0) Setting default file store to GridFS 
I20210126-13:36:37.040(0) Starting Email Intercepter... 
I20210126-13:36:39.486(0) Loaded the Apps Framework and loaded a total of 0 Apps! 
I20210126-13:36:39.954(0) Updating process.env.MAIL_URL 
I20210126-13:36:40.515(0) Using GridFS for custom sounds storage 
I20210126-13:36:40.524(0) Using GridFS for custom emoji storage 
I20210126-13:36:46.875(0) ➔ System ➔ startup 
I20210126-13:36:46.876(0) ➔ +-----------------------------------------------+ 
I20210126-13:36:46.877(0) ➔ |                 SERVER RUNNING                | 
I20210126-13:36:46.878(0) ➔ +-----------------------------------------------+ 
I20210126-13:36:46.879(0) ➔ |                                               | 
I20210126-13:36:46.880(0) ➔ |  Rocket.Chat Version: 3.10.4                  | 
I20210126-13:36:46.881(0) ➔ |       NodeJS Version: 12.18.4 - x64           | 
I20210126-13:36:46.882(0) ➔ |      MongoDB Version: 4.0.22                  | 
I20210126-13:36:46.883(0) ➔ |       MongoDB Engine: mmapv1                  | 
I20210126-13:36:46.884(0) ➔ |             Platform: linux                   | 
I20210126-13:36:46.884(0) ➔ |         Process Port: 4555                    | 
I20210126-13:36:46.885(0) ➔ |             Site URL: http://localhost:4555/  | 
I20210126-13:36:46.886(0) ➔ |     ReplicaSet OpLog: Enabled                 | 
I20210126-13:36:46.887(0) ➔ |          Commit Hash: a4fab4c2d5              | 
I20210126-13:36:46.888(0) ➔ |        Commit Branch: HEAD                    | 
I20210126-13:36:46.889(0) ➔ |                                               | 
I20210126-13:36:46.890(0) ➔ +-----------------------------------------------+ 
I20210126-13:40:28.729(0) Exception while invoking method getUserRoles Error: Invalid user [error-invalid-user]     at MethodInvocation.getUserRoles (app/lib/server/methods/getUserRoles.js:9:10)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:40:28.740(0) Exception while invoking method listCustomUserStatus Error: Invalid user [error-invalid-user]     at MethodInvocation.listCustomUserStatus (app/user-status/server/methods/listCustomUserStatus.js:9:10)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:40:46.910(0) Exception while invoking method login Error: User not found [403]     at handleError (packages/accounts-password/password_server.js:110:17)     at MethodInvocation.<anonymous> (packages/accounts-password/password_server.js:313:5)     at packages/accounts-base/accounts_server.js:475:31     at tryLoginMethod (packages/accounts-base/accounts_server.js:1309:14)     at AccountsServer._runLoginHandlers (packages/accounts-base/accounts_server.js:473:22)     at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:7:35)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:533:31)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:41:36.524(0) Unloaded the Apps Framework. 
I20210126-13:42:22.820(0) Exception while invoking method login Error: User not found [403]     at handleError (packages/accounts-password/password_server.js:110:17)     at MethodInvocation.<anonymous> (packages/accounts-password/password_server.js:313:5)     at packages/accounts-base/accounts_server.js:475:31     at tryLoginMethod (packages/accounts-base/accounts_server.js:1309:14)     at AccountsServer._runLoginHandlers (packages/accounts-base/accounts_server.js:473:22)     at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:7:35)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:533:31)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:42:32.443(0) Exception while invoking method login Error: User not found [403]     at handleError (packages/accounts-password/password_server.js:110:17)     at MethodInvocation.<anonymous> (packages/accounts-password/password_server.js:313:5)     at packages/accounts-base/accounts_server.js:475:31     at tryLoginMethod (packages/accounts-base/accounts_server.js:1309:14)     at AccountsServer._runLoginHandlers (packages/accounts-base/accounts_server.js:473:22)     at AccountsServer.Accounts._runLoginHandlers (app/lib/server/lib/loginErrorMessageOverride.js:7:35)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:533:31)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:42:59.093(0) Updating process.env.MAIL_URL 
I20210126-13:42:59.105(0) Starting Email Intercepter... 
I20210126-13:42:59.506(0) Loaded the Apps Framework and loaded a total of 0 Apps! 
I20210126-13:43:06.307(0) LocalStore: store created at  
I20210126-13:43:06.309(0) LocalStore: store created at  
I20210126-13:43:06.310(0) LocalStore: store created at  
I20210126-13:43:07.206(0) Starting Email Intercepter... 
I20210126-13:43:07.208(0) Updating process.env.MAIL_URL 
I20210126-13:43:09.312(0) Setting default file store to GridFS 
I20210126-13:45:37.879(0) Exception while invoking method loadMissedMessages Error: Invalid user [error-invalid-user]     at MethodInvocation.loadMissedMessages (server/methods/loadMissedMessages.js:13:10)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 
I20210126-13:45:37.913(0) Exception while invoking method loadMissedMessages Error: Invalid user [error-invalid-user]     at MethodInvocation.loadMissedMessages (server/methods/loadMissedMessages.js:13:10)     at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:76:34)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/root/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:262:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 

Start by comparing the MongoDB data size and Rocket.Chat startup logs around each reset, using the reported deployment versions and the entries from app/lib/server/methods/getUserRoles.js and app/user-status/server/methods/listCustomUserStatus.js as context. Done requires a reproducible cause for the lost users and settings, or evidence that the persistence issue is fixed and verified after restart or extended usage.