Implement chunked and parallel Upload in serve WebDAV .
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 59.8k
- Forks
- 5.4k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 48
Description
The associated forum post URL from https://forum.rclone.org
https://forum.rclone.org/t/add-general-webdav-serve-chunked-upload-support/46089
What is your current rclone version (output from rclone version)?
v1.66.0.0
What problem are you are trying to solve?
rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.
How do you think rclone should be changed to solve that?
rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.
Add support for sending chunked uploads to the webdav backend, without the remote file in webdav being chunked/fragmented.
Add support for receiving uploads in chunks by the webdav server rclone serves webdav with vendor = other and type = webdav, merging the fragments into a single file.
Currently it is only possible to send uploads with chunker mode:
[chunker]
type = chunker
remote = Remote_Webdav_Server:
chunk_size=100M
However, this method creates several parts of files on the webdav backend server, the files are not merged.
What I'm proposing is that webdav works normally, however with a command --webdav-upload-chunk-size or --webdav-upload-cutoff the upload would be done in parts, and received in parts by the webdav backend works, and written normally in the filesystem without creating file parts.
--webdav-upload-cutoff=100M activates the sending of the file in chunks over the network, through Accept-Ranges: bytes
I also propose the option of sending multiple parts in parallel, speeding up file upload. --webdav-upload-parallels
The default would be 1 part at a time.
--webdav-upload-parallels=5 would enable 5-part parallel upload.
Implementation already present in part in nextcloud webdav, related #3503
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing related issue #3503 and the existing Nextcloud WebDAV implementation mentioned in the issue, then trace the rclone serve webdav entry point and the chunker configuration example. Done means WebDAV uploads can be sent and received in chunks, merged into one remote file, with the proposed cutoff and parallelism options supported without fragment files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100