rclone / rclone/rclone

Rclone Dedupe does not pass all of Google drive commands through the pacer

Open
#3,199 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Remote: Drive
Dominant language
Go
Stars
59.8k
Forks
5.4k
Avg merge
2d 14h
Merged PRs (30d)
48

Description

What is the problem you are having with rclone?

Not all commands are passed through the pacer for Google Drive
Normally this task would take around 50 mins for me
But if this error occurs it stops after 10 mins

What is your rclone version (output from rclone version)

1.47
Currently testing on v1.47.0-070-g8c550363-beta-windows-amd64

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 x64 1809

Which cloud storage system are you using? (eg Google Drive)

Google Drive with Crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

C:\rclone\rclone.exe dedupe --tpslimit 1 --checkers 4 --fast-list --dedupe-mode rename --progress --log-file "C:\MirrorD Log\Rclone DeDupe %DATEANDTIME%.log" --log-level INFO cryptGDrive:

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Will post this when I've got it reproduced. It doesn't always happen, just 15% of the time its run

So for the moment with just INFO level logging
2019/05/15 11:10:51 INFO : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/05/15 11:12:37 INFO : Encrypted drive 'cryptGDrive:': Looking for duplicates using rename mode.
2019/05/15 11:21:45 Failed to dedupe: find duplicate dirs: couldn't list directory: googleapi: Error 429: Rate Limit Exceeded, rateLimitExceeded

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported rclone dedupe --tpslimit 1 ... cryptGDrive: command with verbose logging, since the issue provides no file or test location. Trace the Google Drive requests made during deduplication and compare them with the pacer behavior; done means the run no longer intermittently fails with a 429 rate-limit error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, google-cloud
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.