RocketChat / RocketChat/Rocket.Chat

File Upload bigger than 2GB fails

Open
#22,513 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
46.1k
Forks
13.9k
Avg merge
3d 3h
Merged PRs (30d)
130

Description

Description:

When uploading files bigger than 2GB, an error occurs.

Steps to reproduce:
  1. Upload a file that is bigger than 2GB
Expected behavior:

The file should be uploaded successfully

Actual behavior:

An error occurs:
image

Server Setup Information:
  • Version of Rocket.Chat Server: 3.15.1
  • Operating System: Ubuntu 20 64 bit
  • Deployment Method: docker
  • Number of Running Instances: 1
  • DB Replicaset Oplog: unknown
  • NodeJS Version: 12.22.1
  • MongoDB Version: 4.0
Client Setup Information
  • Desktop App or Browser Version: Firefox 89.0.1
  • Operating System: Ubuntu 20 64 bit
Additional context

I look at the code and it seems that files are not streamed from start to finish. They are fed into buffers (therefore files end up completely in the RAM) for processing things like removing metadata and filtering the file data. I read that Node on 64 bit only supports 2GB buffers, so I think it is caused by that.
The error logs seem to support that theory:

Relevant logs:

Sever Log
Exception in callback of async function: RangeError [ERR_INVALID_OPT_VALUE] [ERR_INVALID_OPT_VALUE]: The value "2147483648" is invalid for option "size"
at Function.allocUnsafe (buffer.js:383:3)
at Function.concat (buffer.js:564:25)
at FileStream. (app/api/server/v1/rooms.js:82:83)
at FileStream.emit (events.js:326:22)
at FileStream.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'ERR_INVALID_OPT_VALUE'

Client JS Console:
image

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in app/api/server/v1/rooms.js at the FileStream callback around line 82, where the log shows Buffer.concat failing for a 2GB allocation. Reproduce the upload with a file larger than 2GB and trace how the file data is processed. Done means the upload completes successfully without the ERR_INVALID_OPT_VALUE failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.