rclone / rclone/rclone

Implement chunked and parallel Upload in serve WebDAV .

Open
#7,860 0 comments 16 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Remote: WebDAV
Dominant language
Go
Stars
59.8k
Forks
5.4k
Avg merge
2d 14h
Merged PRs (30d)
48

Description

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/add-general-webdav-serve-chunked-upload-support/46089

What is your current rclone version (output from rclone version)?

v1.66.0.0

What problem are you are trying to solve?

rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.

How do you think rclone should be changed to solve that?

rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.

Add support for sending chunked uploads to the webdav backend, without the remote file in webdav being chunked/fragmented.

Add support for receiving uploads in chunks by the webdav server rclone serves webdav with vendor = other and type = webdav, merging the fragments into a single file.

Currently it is only possible to send uploads with chunker mode:

[chunker]
type = chunker
remote = Remote_Webdav_Server:
chunk_size=100M

However, this method creates several parts of files on the webdav backend server, the files are not merged.

What I'm proposing is that webdav works normally, however with a command --webdav-upload-chunk-size or --webdav-upload-cutoff the upload would be done in parts, and received in parts by the webdav backend works, and written normally in the filesystem without creating file parts.
--webdav-upload-cutoff=100M activates the sending of the file in chunks over the network, through Accept-Ranges: bytes

I also propose the option of sending multiple parts in parallel, speeding up file upload. --webdav-upload-parallels
The default would be 1 part at a time.
--webdav-upload-parallels=5 would enable 5-part parallel upload.

Implementation already present in part in nextcloud webdav, related #3503

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing related issue #3503 and the existing Nextcloud WebDAV implementation mentioned in the issue, then trace the rclone serve webdav entry point and the chunker configuration example. Done means WebDAV uploads can be sent and received in chunks, merged into one remote file, with the proposed cutoff and parallelism options supported without fragment files.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.