Messages sending out of order (or, frequently, not at all)

Description:

Messages are displaying out of order, if they display at all.

Steps to reproduce:

1. Send a private message.
2. Anticipate it showing on the next line below.
3. Observe it showing up several lines above.

Expected behavior:

Send a private message.
Observe it showing on the next line below.

Actual behavior:

Send a private message.
Observe it showing up several lines above.

Server Setup Information:

• Version of Rocket.Chat Server: 0.74.3
• Operating System: Docker\Kubernetes
• Deployment Method: Kubernetes
• Number of Running Instances: 4
• DB Replicaset Oplog: Yes
• NodeJS Version: Baked into image
• MongoDB Version: 3.6

Additional context

We've had nothing but issues with our Rocketchat install. Notifications not being sent, having to constantly reload without cache to be able to see messages, messages that don't show up for weeks...

Some of our config:

    spec:
      containers:
      - env:
        - name: MAIL_URL
          value: smtp://ehs-*****.****.ad
        - name: MONGO_OPLOG_URL
          value: mongodb://10.***.***.41:27017,10.***.***.42:27017,10.***.***.43:27017,10.***.***.44:27017/local?replicaSet=rs0
        - name: MONGO_URL
          value: mongodb://10.***.***.41:27017,10.***.***.42:27017,10.***.***.43:27017,10.***.***.44:27017/rocketchat?replicaSet=rs0&readPreference=nearest&w=majority
        - name: PORT
          value: "3000"
        - name: ROOT_URL
          value: http://ehs-******.******.ad
        - name: INSTANCE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        image: rocketchat/rocket.chat:latest
        imagePullPolicy: IfNotPresent
        name: rocketchat
        ports:
        - containerPort: 3000
          hostPort: 3000
          name: 3000tcp30000
          protocol: TCP
        - containerPort: 3000
          name: 3000tcp803
          protocol: TCP
        - containerPort: 9100
          name: 9100tcp2
          protocol: TCP

Port 3000 to 3000 is a host port to allow inter-server communication.
Port 3000 to 80 is a LoadBalancer port to allow load balanced access from the clients (and to enable HA).
Port 9100 is a cluster IP port to allow Prometheus scrapes for metrics.

Relevant logs:

We are only consistently getting a single type of error message in our logs since disabling DDP rate limiting (which prevented us from using Rocketchat at all):

Error in oplog callback TypeError: Cannot read property 'u' of undefined
at BaseDb.RocketChat.models.Subscriptions.on (server/publications/subscription.js:84:50)
at emitOne (events.js:116:13)
at BaseDb.emit (events.js:211:7)
at BaseDb.processOplogRecord (packages/rocketchat:models/server/models/_BaseDb.js:143:9)
at packages/mongo/oplog_tailing.js:105:7
at runWithEnvironment (packages/meteor.js:1356:24)
at Object.callback (packages/meteor.js:1369:14)
at packages/ddp-server/crossbar.js:114:36
at Array.forEach (<anonymous>)
at Function._.each._.forEach (packages/underscore.js:139:11)
at DDPServer._Crossbar.fire (packages/ddp-server/crossbar.js:112:7)
at handleDoc (packages/mongo/oplog_tailing.js:311:24)
at packages/mongo/oplog_tailing.js:337:11
at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1304:12)
at packages/meteor.js:620:25
at runWithEnvironment (packages/meteor.js:1356:24)

Start with server/publications/subscription.js at line 84 and packages/rocketchat:models/server/models/_BaseDb.js at line 143, then trace the MongoDB oplog callback and related subscription handling. Reproduce the missing or out-of-order private messages in the Kubernetes multi-instance setup and determine whether the logged undefined-user error is involved. Done means the error and message-delivery behavior are resolved with coverage for the regression.