rclone / rclone/rclone

drime: support for single-part presigned uploads

Open
#9,750 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Remote: Drime
Dominant language
Go
Stars
59.8k
Forks
5.4k
Avg merge
2d 14h
Merged PRs (30d)
48

Description

Before you start
  • I have searched the forum and the existing issues and this hasn't already been requested.
  • I have checked the latest beta and this feature doesn't already exist.
Associated forum post URL

No response

rclone version

rclone v1.75.0

  • os/version: arch (64 bit)
  • os/kernel: 7.1.6-arch1-1 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.26.5
  • go/linking: dynamic
  • go/tags: none
What problem are you trying to solve?

Drime backend is very slow, which becomes obvious when you try to upload thousands of small files. Drime is aware of this issue but they can't improve it for now. I've been looking into the upload options available in the Drime API and which ones Rclone actually uses.

Drime offers three upload options: Default, Presigned URL (S3) and Multipart Upload (S3). Rclone uses default and multipart. With the default upload, Rclone sends the entire file to Drime, so the backend has to read the whole thing, calculate the checksum and then upload it to Cloudflare. This puts a lot of strain on the backend. As we've seen this year, they can't handle large files which is why they changed their recommendation to only use this for files under 5MB (it used to be 200MB). The only advantage is that it's very simple and only requires one API call.

Multipart is an option that removes the need for the backend to receive the file, making it ideal for large files. However, it requires (IIUC) four API calls and more database writes (the database is also slow). And it feels like they have some kind of rate limit for API calls.

How do you think rclone should be changed to solve that?

Drime has another option that Rclone doesn't support yet: https://docs.drime.cloud/api-reference/uploads/presign-url. This only requires two simpler API calls. I assume it's a newer feature which is why Rclone doesn't have it. They recommend it for files under 5MB but it should work for larger ones too. It's basically like multipart but without having to split the file.

Since Rclone already has an option (--drime-upload-cutoff) to decide when to use multipart, I think you could add a new flag, something like --drime-prefer-presigned-uploads. That would let the user choose what to use for uploads below the cutoff.

I can't write the code myself but I can test any binary you give me.

Getting involved
  • I'm willing to help implement, test or fund this feature.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Drime backend's existing default and multipart upload paths and the handling for --drime-upload-cutoff. Read the linked Drime presigned-upload API documentation and compare its two-call flow with the current paths. Done means an optional preference can select presigned uploads below the cutoff and the resulting Drime uploads work correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.