Make parts assembly a non blocking request
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
My Setup
- I have Raspberry PI 3B 1GB RAM
- 32 GB SDCard with moderate read/write speeds
- 4TB external hard drive formatted to BTRFS
- Fresh NextCloudPi with default settings, except that Data dir has been moved to HDD
- 2,5 GB video file, that I want to upload through NextCloud Web UI
Observed behavior
- I go to needed folder in Web UI, click "Upload file" button and select my video
- File is split in to chunks and uploaded one chunk at a time. This process is OK.
- Web UI asks server to assemble chunks in to a single file, internally this is a blocking request
- Chunks and resulting file are on the same HDD, so my drive has to constantly switch from read to write mode and reposition reading head each time
- Assembly of the file takes more than 10 minutes
- Some timeouts end before file assembly is complete and Web UI displays an error
- File upload fails
My fix
I have enabled 'localstorage.allowsymlinks' option and symlinked /media/HDD/ncdata/data/myuser/uploads to /tmp on SDCard. In general, my SDCard is as fast as my HDD. But during file assembly, both storage devices can work in their optimal modes, so assembly time is cut significantly. In most cases I get 2x speedup. This is enough to successfully upload my 2,5 GB video file.
Conclusions
- No matter how fast your hardware is, with large enough file, chunk assembly will timeout. So, please, make related calls nonblocking.
- Moving ./data/user/uploads folder to a separate drive can significantly speedup NextCloud instances, that use HDDs as primary storage. Please, add such option to the config.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not identify any files, tests, or entry points. Start by tracing the server-side chunk-assembly request and its timeout behavior; done means large uploads assemble asynchronously without request timeouts and the proposed uploads-storage configuration is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100