rclone / rclone/rclone

Union: for every single transfer rclone listing all the remotes

Open
#4,879 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

change detection Remote: Union
Dominant language
Go
Stars
59.8k
Forks
5.4k
Avg merge
2d 14h
Merged PRs (30d)
48

Description

[onedriveunion]
type = union
upstreams = [20 onedrive business here...]
cache_time = 604800

Command:

rclone sync onedriveunion: /path/to/store --check-first --transfers=8 -P

Issue: The problem is that each time a file needs to transfer, Rclone checking all the remotes for the file, until the file is found. So, in every single transfer rclone listing all the remotes again and again. It's not remembering the location even when --check-first is applied.

We need some caching here. Otherwise it's practically impossible to use Union.

Here is what happening:

1) File1 needs to transfer.
2) Searching File1 for all remotes. ------------> This is the problem.
3) File1 found on OneDrive5
3) File1 transfer complete.

1) File2 needs to transfer.
2) Searching File2 for all remotes.  ------------> This is the problem.
3) File2 found on OneDrive8
3) File2 transfer complete.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how Union handles remote listings during rclone sync, particularly with --check-first, and identify where each file lookup repeats the upstream searches. Done means the lookup location or result is cached so repeated transfers do not list every remote again; add or run coverage for a file found on different upstream remotes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.