General instability since version 6.2.6

Description:

I am running a RocketChat server since early 2019. It's been working fine until I upgraded it from 5.1.4 to 6.2.6 back in June 2023. I have since updated to 6.4.2, but the instability persists.

Symptoms include:

• After pressing Enter to send a message, the message does not appear in the chat (channels, threads DM alike). It's not possible to edit it with "arrow up". It's just lost.
• Opening a channel doesn't mark it as read. Sometime forcing "mark as read" from the sidebar does not work.
• Uploading a picture or file fails silently, it does not show the interface to change the description before posting.
• Channels / threads stop updating.

All of these happen without explicit errors, and the only solution is to fully reload the page.

I have tried the following improvements:

• Scaling the VM, which now has 4 cores and 24 GiB RAM
• Replace Apache with Nginx as reverse-proxy
• Running 3 instances of RocketChat (Systemd services) following this doc
• Moving to faster storage (the VM runs off a NVMe SSD local to the hypervisor)

And the following troubleshooting:

• Enable Prometheus to export the app metrics to our Grafana instance.
• Check logs to look for errors.
• Check monitoring from the VM / network itself.
• Check the browser console to look for errors.

The issue is still there.

It's a rather small instance. We have about 25 active users and another 25 away during the day. The whole MongoDB weighs 60 GiB (there's years of stuff in there).

Load on the VM is minimal:

Load on the app itself is minimal:

App monitoring show it's responsive (Meteor Methods Total Time):

The "Messages round trip" value seems of high, but I'm unsure about what this is exactly since this value is missing from the doc.

These values (Callback: API_Embed and Hook: AfterSaveMessage) also seem high

Steps to reproduce:

Issue appears randomly but often since the upgrade to 6.x.x.

Expected behavior:

RocketChat is as stable as it was on version 5.x.

Actual behavior:

Server Setup Information:

• Version of Rocket.Chat Server: 6.4.2
• Operating System: Debian 11.8
• Deployment Method: Tar
• Number of Running Instances: 3 instances on same 4-core VM
• DB Replicaset Oplog:
• NodeJS Version: 14.21.3
• MongoDB Version: 5.0.18 / wiredTiger (oplog Enabled)

Client Setup Information

• Desktop App or Browser Version: Reproduced in recent version of Chrome and Firefox, as well as native apps (Linux, Android, IOS).
• Operating System:

Additional context

The issue has been noticed by most users regardless of OS, browser, physical location...

Relevant logs:

No source file, test, or entry point is identified. Start by reproducing the failures across the listed clients while checking browser-console and server logs, then compare behavior with version 5.1.4 and inspect the reported metrics; done means identifying and addressing the cause so sending, read state, uploads, and live updates work without a full reload.