drime: support for single-part presigned uploads
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 59.8k
- Forks
- 5.4k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 48
Description
Before you start
- I have searched the forum and the existing issues and this hasn't already been requested.
- I have checked the latest beta and this feature doesn't already exist.
Associated forum post URL
No response
rclone version
rclone v1.75.0
- os/version: arch (64 bit)
- os/kernel: 7.1.6-arch1-1 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.5
- go/linking: dynamic
- go/tags: none
What problem are you trying to solve?
Drime backend is very slow, which becomes obvious when you try to upload thousands of small files. Drime is aware of this issue but they can't improve it for now. I've been looking into the upload options available in the Drime API and which ones Rclone actually uses.
Drime offers three upload options: Default, Presigned URL (S3) and Multipart Upload (S3). Rclone uses default and multipart. With the default upload, Rclone sends the entire file to Drime, so the backend has to read the whole thing, calculate the checksum and then upload it to Cloudflare. This puts a lot of strain on the backend. As we've seen this year, they can't handle large files which is why they changed their recommendation to only use this for files under 5MB (it used to be 200MB). The only advantage is that it's very simple and only requires one API call.
Multipart is an option that removes the need for the backend to receive the file, making it ideal for large files. However, it requires (IIUC) four API calls and more database writes (the database is also slow). And it feels like they have some kind of rate limit for API calls.
How do you think rclone should be changed to solve that?
Drime has another option that Rclone doesn't support yet: https://docs.drime.cloud/api-reference/uploads/presign-url. This only requires two simpler API calls. I assume it's a newer feature which is why Rclone doesn't have it. They recommend it for files under 5MB but it should work for larger ones too. It's basically like multipart but without having to split the file.
Since Rclone already has an option (--drime-upload-cutoff) to decide when to use multipart, I think you could add a new flag, something like --drime-prefer-presigned-uploads. That would let the user choose what to use for uploads below the cutoff.
I can't write the code myself but I can test any binary you give me.
Getting involved
- I'm willing to help implement, test or fund this feature.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Drime backend's existing default and multipart upload paths and the handling for --drime-upload-cutoff. Read the linked Drime presigned-upload API documentation and compare its two-call flow with the current paths. Done means an optional preference can select presigned uploads below the cutoff and the resulting Drime uploads work correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100