RocketChat / RocketChat/Rocket.Chat

Large "Download My Data" export crashes RC

Open
#19,737 5 comments 0 reactions 1 assignee View on GitHub

@gabrieleiro is already working on this.

Since Apr 27, 2021.

  • #21898 by @gabrieleiro — closed without merging
Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

When "Download My Data" is initiated for a user who has an excessive amount of historical data, the RC service will crash while trying to compile an export archive.

Steps to reproduce:
  1. Click on "Download My Data" for a user with a large amount of historical data in the system.
  2. Observe that eventually the system will grind to a halt.
Expected behavior:

The archive should be produced and a link should be emailed to the user.

Actual behavior:

The export job tries to gather all historical data into an archive, but the volume of data eventually brings down the system. My assumption is that this is due to the usage of /tmp or tmpfs as the workspace for compiling "Download My Data" archives. I have not tested it yet, but I imagine a channel export of a channel with excessive amounts of data (assets, etc.) would also produce a problem.

Server Setup Information:
  • Version of Rocket.Chat Server: 3.6.0
  • Operating System: RedHat
  • Deployment Method: tar file
  • Number of Running Instances: 3
  • DB Replicaset Oplog:
  • NodeJS Version:
  • MongoDB Version:
Client Setup Information
  • Desktop App or Browser Version: Chrome
  • Operating System:
Additional context

Our sever has 2.0G of space allocated for tmpfs. Ideally, the temporary area used by the export functions to compile archives of exported data would take place in a much larger area. I'm not an expert in tmpfs, so please correct me if I am wrong, but I believe the max of what you can allocate to it is limited by your server's RAM, and that's really not enough to support one user with a lot of data, much less any concurrent exports that may take place.

Relevant logs:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.