Rclone failed to process files whose filename or path contains "‛"

The associated forum post URL from https://forum.rclone.org

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

rclone v1.58.0

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

debian 11.2 (64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

SFTP and local filesystem

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone copy /tmp/test /tmp/test2
rclone copy /tmp/test remote:test

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2022/04/11 12:14:16 DEBUG : rclone: Version "v1.58.0" starting with parameters ["rclone" "copy" "/tmp/test" "/tmp/test2" "-vvv"]
2022/04/11 12:14:16 DEBUG : Creating backend with remote "/tmp/test"
2022/04/11 12:14:16 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2022/04/11 12:14:16 DEBUG : Creating backend with remote "/tmp/test2"
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for checks to finish
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for transfers to finish
2022/04/11 12:14:16 ERROR : test‛‛s: Failed to copy: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 DEBUG : tests: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2022/04/11 12:14:16 INFO : tests: Copied (new)
2022/04/11 12:14:16 ERROR : Attempt 1/3 failed with 1 errors and: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 ERROR : test‛‛s: Failed to copy: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for checks to finish
2022/04/11 12:14:16 DEBUG : tests: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/04/11 12:14:16 DEBUG : tests: Unchanged skipping
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for transfers to finish
2022/04/11 12:14:16 ERROR : Attempt 2/3 failed with 1 errors and: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 ERROR : test‛‛s: Failed to copy: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 DEBUG : tests: Size and modification time the same (differ by 0s, within tolerance 1ns)
2022/04/11 12:14:16 DEBUG : tests: Unchanged skipping
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for checks to finish
2022/04/11 12:14:16 DEBUG : Local file system at /tmp/test2: Waiting for transfers to finish
2022/04/11 12:14:16 ERROR : Attempt 3/3 failed with 1 errors and: failed to open source object: open /tmp/test/test‛‛s: no such file or directory
2022/04/11 12:14:16 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Errors: 1 (retrying may help)
Checks: 2 / 2, 100%
Transferred: 1 / 1, 100%
Elapsed time: 0.1s

2022/04/11 12:14:16 DEBUG : 4 go routines active
2022/04/11 12:14:16 Failed to copy: failed to open source object: open /tmp/test/test‛‛s: no such file or directory

Bug Description

If the source filename contain a special character ‛ (Unicode U+201B), rclone will try processing the file with double ‛ which of course doesn't exist.

In another scenario, I tried synchronizing files from a remote server to local computer through sftp. The local computer only has the directory structure where some folders' names contain ‛. When copying, rclone showed notices like "Duplicate directory found in destination - ignoring", created those folders with double "‛" and considered as non-existent even when I ran rclone copy again after a successful transfer.

rclone dedupe will also fail if I use it at two folders with names containing single ‛ and double ‛ respectively. It prompts "Can't have duplicate names here. Perhaps you wanted --by-hash ? Continuing anyway." and "error listing: directory not found"

Procedure to reproduce this bug:
mkdir /tmp/test && touch /tmp/test/test‛s && touch /tmp/test/tests
rclone copy /tmp/test /tmp/test2 -vv

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Start with the rclone copy command and local filesystem path in the reproduction, then compare behavior for filenames containing U+201B and ordinary filenames. Reproduce the failure with the provided mkdir and touch commands, and verify that copy, SFTP-to-local transfer, and deduplication no longer treat the character as a different or duplicated name.