nextcloud / nextcloud/server

Make parts assembly a non blocking request

Open
#31,259 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage enhancement feature: dav hotspot: file transfer performance performance 🚀
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

My Setup
  • I have Raspberry PI 3B 1GB RAM
  • 32 GB SDCard with moderate read/write speeds
  • 4TB external hard drive formatted to BTRFS
  • Fresh NextCloudPi with default settings, except that Data dir has been moved to HDD
  • 2,5 GB video file, that I want to upload through NextCloud Web UI
Observed behavior
  1. I go to needed folder in Web UI, click "Upload file" button and select my video
  2. File is split in to chunks and uploaded one chunk at a time. This process is OK.
  3. Web UI asks server to assemble chunks in to a single file, internally this is a blocking request
  4. Chunks and resulting file are on the same HDD, so my drive has to constantly switch from read to write mode and reposition reading head each time
  5. Assembly of the file takes more than 10 minutes
  6. Some timeouts end before file assembly is complete and Web UI displays an error
  7. File upload fails
My fix

I have enabled 'localstorage.allowsymlinks' option and symlinked /media/HDD/ncdata/data/myuser/uploads to /tmp on SDCard. In general, my SDCard is as fast as my HDD. But during file assembly, both storage devices can work in their optimal modes, so assembly time is cut significantly. In most cases I get 2x speedup. This is enough to successfully upload my 2,5 GB video file.

Conclusions
  • No matter how fast your hardware is, with large enough file, chunk assembly will timeout. So, please, make related calls nonblocking.
  • Moving ./data/user/uploads folder to a separate drive can significantly speedup NextCloud instances, that use HDDs as primary storage. Please, add such option to the config.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue does not identify any files, tests, or entry points. Start by tracing the server-side chunk-assembly request and its timeout behavior; done means large uploads assemble asynchronously without request timeouts and the proposed uploads-storage configuration is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.