rclone / rclone/rclone

webdav: dCache: add support for modtime syncing

Open
#8,593 19 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement Remote: WebDAV
Dominant language
Go
Stars
59.8k
Forks
5.4k
Avg merge
2d 14h
Merged PRs (30d)
48

Description

What is the problem you are having with rclone?

The sync command does not update a remote file when the local file's contents have changed but the file size is the same when using dCache.

This is because dCache does not support modtimes, however recent changes means that it supports the X-OC-Mtime header and rclone should be using this.

What is your rclone version (output from rclone version)

I've tried this with the version from Debian 12 (bookworm)

paul@monkeywrench:~$ /usr/bin/rclone --version
rclone v1.60.1-DEV
- os/version: debian 12.11 (64 bit)
- os/kernel: 6.1.0-35-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.8
- go/linking: dynamic
- go/tags: none
paul@monkeywrench:~$ dpkg -l rclone
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version          Architecture Description
+++-==============-================-============-==================================
ii  rclone         1.60.1+dfsg-2+b5 amd64        rsync for commercial cloud storage
paul@monkeywrench:~$ 

I've also used selfupdate to tried this with the latest version (v1.69.3) and the latest beta (v1.70.0-beta.8772.dda6a863e).

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Debian amd64 architecture (64 bit).

Which cloud storage system are you using? (e.g. Google Drive)

WebDAV, targeting a dCache instance.

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)
rclone sync test-file prometheus:/Users/paul/target

The prometheus storage has the following configuration (in ~/.config/rclone/rclone.conf`)

[prometheus]
type = webdav
url = https://prometheus.desy.de:2443/
vendor = other
bearer_token_command = oidc-token EGI-CHECKIN-DEMO
A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Here's a walk-through, demonstrating the problem.

First the remote file (if any) is removed

paul@monkeywrench:~$ rclone delete prometheus:/Users/paul/target/test-file
paul@monkeywrench:~$ rclone cat prometheus:/Users/paul/target/test-file
2025/06/10 13:03:19 ERROR : error listing: directory not found
2025/06/10 13:03:19 NOTICE: Failed to cat with 2 errors: last error was: directory not found
paul@monkeywrench:~$ 

Next, a file test-file is created with FOO as its content. This is synced to the remote storage and the remote content is verified. This works as expected.

paul@monkeywrench:~$ echo FOO > test-file
paul@monkeywrench:~$ rclone sync -vv test-file prometheus:/Users/paul/target
2025/06/10 13:04:31 DEBUG : rclone: Version "v1.70.0-beta.8772.dda6a863e" starting with parameters ["rclone" "sync" "-vv" "test-file" "prometheus:/Users/paul/target"]
2025/06/10 13:04:31 DEBUG : Creating backend with remote "test-file"
2025/06/10 13:04:31 DEBUG : Using config file from "/home/paul/.config/rclone/rclone.conf"
2025/06/10 13:04:31 DEBUG : fs cache: renaming child cache item "test-file" to be canonical for parent "/home/paul"
2025/06/10 13:04:31 DEBUG : Creating backend with remote "prometheus:/Users/paul/target"
2025/06/10 13:04:31 DEBUG : found headers: 
2025/06/10 13:04:31 DEBUG : fs cache: renaming cache item "prometheus:/Users/paul/target" to be canonical "prometheus:Users/paul/target"
2025/06/10 13:04:31 DEBUG : test-file: Need to transfer - File not found at Destination
2025/06/10 13:04:32 DEBUG : test-file: Update will use the normal upload strategy (no chunks)
2025/06/10 13:04:32 INFO  : test-file: Copied (new)
2025/06/10 13:04:32 INFO  : 
Transferred:              4 B / 4 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2025/06/10 13:04:32 DEBUG : 6 go routines active
paul@monkeywrench:~$ rclone cat prometheus:/Users/paul/target/test-file
FOO
paul@monkeywrench:~$ 

Next, the content of test-file is updated to contain new content. We then attempt to sync this with the remote content. We show that the remote content has not been updated. The output suggests that only the file's size is used to determine whether the remote file needs to be updated.

paul@monkeywrench:~$ echo BAR > test-file
paul@monkeywrench:~$ rclone sync -vv test-file prometheus:/Users/paul/target
2025/06/10 13:06:25 DEBUG : rclone: Version "v1.70.0-beta.8772.dda6a863e" starting with parameters ["rclone" "sync" "-vv" "test-file" "prometheus:/Users/paul/target"]
2025/06/10 13:06:25 DEBUG : Creating backend with remote "test-file"
2025/06/10 13:06:25 DEBUG : Using config file from "/home/paul/.config/rclone/rclone.conf"
2025/06/10 13:06:25 DEBUG : fs cache: renaming child cache item "test-file" to be canonical for parent "/home/paul"
2025/06/10 13:06:25 DEBUG : Creating backend with remote "prometheus:/Users/paul/target"
2025/06/10 13:06:25 DEBUG : found headers: 
2025/06/10 13:06:25 DEBUG : fs cache: renaming cache item "prometheus:/Users/paul/target" to be canonical "prometheus:Users/paul/target"
2025/06/10 13:06:25 DEBUG : test-file: Sizes identical
2025/06/10 13:06:25 DEBUG : test-file: Unchanged skipping
2025/06/10 13:06:25 INFO  : 
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

2025/06/10 13:06:25 DEBUG : 7 go routines active
paul@monkeywrench:~$ rclone cat prometheus:/Users/paul/target/test-file
FOO
paul@monkeywrench:~$ 

Finally, the file test-file is updated so the file size changes (BAZBAZ). The sync command is run and we verify that the remote content has been updated.

paul@monkeywrench:~$ echo BAZBAZ > test-file
paul@monkeywrench:~$ rclone sync -vv test-file prometheus:/Users/paul/target
2025/06/10 13:07:28 DEBUG : rclone: Version "v1.70.0-beta.8772.dda6a863e" starting with parameters ["rclone" "sync" "-vv" "test-file" "prometheus:/Users/paul/target"]
2025/06/10 13:07:28 DEBUG : Creating backend with remote "test-file"
2025/06/10 13:07:28 DEBUG : Using config file from "/home/paul/.config/rclone/rclone.conf"
2025/06/10 13:07:28 DEBUG : fs cache: renaming child cache item "test-file" to be canonical for parent "/home/paul"
2025/06/10 13:07:28 DEBUG : Creating backend with remote "prometheus:/Users/paul/target"
2025/06/10 13:07:28 DEBUG : found headers: 
2025/06/10 13:07:28 DEBUG : fs cache: renaming cache item "prometheus:/Users/paul/target" to be canonical "prometheus:Users/paul/target"
2025/06/10 13:07:28 DEBUG : test-file: Sizes differ (src 7 vs dst 4)
2025/06/10 13:07:28 DEBUG : test-file: Update will use the normal upload strategy (no chunks)
2025/06/10 13:07:28 INFO  : test-file: Copied (replaced existing)
2025/06/10 13:07:28 INFO  : 
Transferred:              7 B / 7 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2025/06/10 13:07:28 DEBUG : 7 go routines active
paul@monkeywrench:~$ rclone cat prometheus:/Users/paul/target/test-file
BAZBAZ
paul@monkeywrench:~$ 
How to use GitHub
  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at rclone sync's WebDAV backend and trace how remote metadata is compared when dCache lacks modtimes. Check how the X-OC-Mtime header is currently handled or can be observed, then reproduce the equal-size FOO/BAR case against dCache. Done means sync detects changed content with equal size and updates the remote file without regressing normal WebDAV behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.