Exception while invoking method getUsersOfRoom

Description:

User lookups are failing.

Steps to reproduce:

Occurs when:

• trying to list members of a channel ("Showing: 0, Online: 0, Total: undefined users")
• showing "Users" page in Administration interface (pulsing placeholder images and a red pop up error on top right "Executor error during find command")
• mentioning users (some users appear in suggestions but not all that should, and continuing to expand name results in "[unidentified]" red error popups in the top right corner of the UI and no match.

Expected behavior:

I would expect relevant users to appear.

Actual behavior:

Server Setup Information:

• Version of Rocket.Chat Server: 3.9.3
• Operating System: Ubuntu 18.04.5 LTS
• Deployment Method: snap
• Number of Running Instances: 1
• DB Replicaset Oplog:
• NodeJS Version:
• MongoDB Version:

Client Setup Information

• Desktop App or Browser Version: Both Rocket.Chat Electron 3.1.1 and Firefox 85.0
• Operating System: MacOS 10.14.6

Additional context

Relevant logs:

Server logs:
I20210203-15:19:30.648(-5) Exception while invoking method getUsersOfRoom MongoError: BSONElement: bad type 114 at Connection.<anonymous> (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/pool.js:451:61) at Connection.emit (events.js:315:20) at Connection.EventEmitter.emit (domain.js:483:12) at processMessage (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:452:10) at Socket.<anonymous> (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/npm-mongo/node_modules/mongodb/lib/core/connection/connection.js:621:15) at Socket.emit (events.js:315:20) at Socket.EventEmitter.emit (domain.js:483:12) at addChunk (_stream_readable.js:295:12) at readableAddChunk (_stream_readable.js:271:9) at Socket.Readable.push (_stream_readable.js:212:10) at TCP.onStreamRead (internal/stream_base_commons.js:186:23) at TCP.callbackTrampoline (internal/async_hooks.js:120:14) => awaited here: at Function.Promise.await (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12) at packages/mongo/mongo_driver.js:1073:14 at /snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 => awaited here: at Promise.await (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at SynchronousCursor._nextObject (packages/mongo/mongo_driver.js:1122:38) at SynchronousCursor.forEach (packages/mongo/mongo_driver.js:1136:22) at SynchronousCursor.map (packages/mongo/mongo_driver.js:1146:10) at SynchronousCursor.fetch (packages/mongo/mongo_driver.js:1170:17) at Cursor.<computed> [as fetch] (packages/mongo/mongo_driver.js:918:44) at findUsers (server/methods/getUsersOfRoom.js:27:6) at server/methods/getUsersOfRoom.js:48:23 at /snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 => awaited here: at Promise.await (/snap/rocketchat-server/1454/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12) at Server.apply (packages/ddp-server/livedata_server.js:1638:22) at Server.call (packages/ddp-server/livedata_server.js:1607:17) at Object.post (app/api/server/v1/misc.js:262:26) at app/api/server/api.js:394:82 at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12) at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39) at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32) at packages/nimble_restivus/lib/route.coffee:59:33 at packages/simple_json-routes.js:98:9

Browser logs:
32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js:426:674008 Object { stack: "r@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:4173\n_livedata_result@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162:17753\nonMessage@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162:20094\nu@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428:125658\ne/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428:125788\nn.bindEnvironment/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:8262\npromise callback*n/t.makeCompatible/n.prototype.then@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:120:1083\ne@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428:125758\nsendMessage@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162:2232\napply@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162:9758\ncall@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162:8296\nt/o.call@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426:873812\nt/</<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426:569093\nn/</<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19108\ne/t._withTemplateInstanceFunc@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:38048\nn/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19059\nt/h._withCurrentView@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:23216\nn@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19007\n_compute@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:2279\nt@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:1410\nn/o.autorun@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:3956\nt/h.View.prototype.autorun@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19190\ne/h.TemplateInstance.prototype.autorun@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:36610\nt/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426:569027\ni/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:34843\ne/t._withTemplateInstanceFunc@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:38048\ni@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:34746\ne/t.prototype.constructView/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:35639\nn@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:20486\nn/o.nonreactive@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:4139\nt/h._fireCallbacks/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:20397\nt/h._withCurrentView@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:23216\nt/h._fireCallbacks@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:20364\nt/h._createView@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:20669\nt/h._materializeView@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:21073\nt@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:15647\ne/h._materializeDOM@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:14881\ne/h._materializeDOM@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:14923\nt@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:21287\nn/o.nonreactive@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:4139\nt@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:21253\nn/</<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19108\ne/t._withTemplateInstanceFunc@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:38048\nn/<@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19059\nt/h._withCurrentView@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:23216\nn@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221:19007\n_compute@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:2279\n_recompute@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:2434\nn/o._runFlush@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146:3443\na@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:3087\nEventListener.handleEvent*r@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:3133\n@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:3398\n@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:3408\n@https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1:9502\n", isClientSafe: true, error: undefined, reason: undefined, details: undefined, message: "[undefined]", errorType: "Meteor.Error" } 32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js:426:569210 Exception in delivering result of invoking 'getUsersOfRoom': TypeError: e is undefined t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 u https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 call https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 bindEnvironment https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 _maybeInvokeCallback https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 receiveResult https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 _livedata_result https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 onMessage https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 u https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428 e https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428 bindEnvironment https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 promise callback*n/t.makeCompatible/n.prototype.then https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:120 e https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:428 sendMessage https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 apply https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 call https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:162 call https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withTemplateInstanceFunc https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withCurrentView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _compute https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 autorun https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 autorun https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 autorun https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:426 i https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withTemplateInstanceFunc https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 i https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 constructView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 nonreactive https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 _fireCallbacks https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withCurrentView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _fireCallbacks https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _createView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _materializeView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _materializeDOM https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _materializeDOM https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 nonreactive https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 t https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withTemplateInstanceFunc https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _withCurrentView https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 n https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:221 _compute https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 _recompute https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 _runFlush https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:146 a https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 r https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 <anonymous> https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 <anonymous> https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 <anonymous> https://rocketchat.gcrc.carleton.ca/32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js?meteor_js_resource=true:1 32054adf510e6e551f0159e28ac4bf9f4d9acdbd.js:1:6771

Start at server/methods/getUsersOfRoom.js, especially findUsers around lines 27 and 48, and inspect the server log's BSONElement error while reproducing member, administration user, and mention lookups. Trace the query and its data assumptions, then verify that all three user lookups return relevant users without the reported exception.