Rclone Dedupe does not pass all of Google drive commands through the pacer
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 59.8k
- Forks
- 5.4k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 48
Description
What is the problem you are having with rclone?
Not all commands are passed through the pacer for Google Drive
Normally this task would take around 50 mins for me
But if this error occurs it stops after 10 mins
What is your rclone version (output from rclone version)
1.47
Currently testing on v1.47.0-070-g8c550363-beta-windows-amd64
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows 10 x64 1809
Which cloud storage system are you using? (eg Google Drive)
Google Drive with Crypt
The command you were trying to run (eg rclone copy /tmp remote:tmp)
C:\rclone\rclone.exe dedupe --tpslimit 1 --checkers 4 --fast-list --dedupe-mode rename --progress --log-file "C:\MirrorD Log\Rclone DeDupe %DATEANDTIME%.log" --log-level INFO cryptGDrive:
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
Will post this when I've got it reproduced. It doesn't always happen, just 15% of the time its run
So for the moment with just INFO level logging
2019/05/15 11:10:51 INFO : Starting HTTP transaction limiter: max 1 transactions/s with burst 1
2019/05/15 11:12:37 INFO : Encrypted drive 'cryptGDrive:': Looking for duplicates using rename mode.
2019/05/15 11:21:45 Failed to dedupe: find duplicate dirs: couldn't list directory: googleapi: Error 429: Rate Limit Exceeded, rateLimitExceeded
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported rclone dedupe --tpslimit 1 ... cryptGDrive: command with verbose logging, since the issue provides no file or test location. Trace the Google Drive requests made during deduplication and compare them with the pacer behavior; done means the run no longer intermittently fails with a 429 rate-limit error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100