s3/fshttp: --timeout only enforces an idle deadline, so a slow trickle (e.g. wedged multipart upload) can hang indefinitely with no error and no retry

What is the problem you are having with rclone?

--timeout does not bound total request/transfer duration -- it only enforces
an idle deadline on the raw connection, reset on every successful Read/Write
of any nonzero byte count (timeoutConn.nudgeDeadline, fs/fshttp/dialer.go).

A peer that drips data slower than a real transfer, but with gaps shorter than
--timeout, can hold a connection open indefinitely. No Read/Write ever
returns an error, so neither --retries nor --low-level-retries ever engage
either -- there is nothing for them to retry.

We hit this in production against a DigitalOcean Spaces bucket: a single
rclone copy multipart part upload hung for ~16 hours. strace//proc
showed an established TCP connection, ongoing (if minimal) syscall activity,
and slowly-increasing CPU ticks across samples -- i.e. not deadlocked, just
never finishing and never erroring. --stats=5m never printed a single
Transferred: line the entire run. The job used --retries=3 --retries-sleep=60s (defaults otherwise) and none of that ever fired.

I've confirmed the mechanism in isolation with a minimal repro rather than
relying on the live incident (which isn't reproducible on demand against a
third-party bucket): see below.

Repro

Branch: https://github.com/halindrome/rclone/tree/repro/timeout-does-not-bound-slow-trickle
(fs/fshttp/stall_repro_test.go, TestTimeoutDoesNotBoundSlowTrickle)

A local net.Listener accepts a request, then writes 20 response bytes one at
a time with a 150ms sleep between each, against a client configured with
--timeout 300ms. Every individual gap (150ms) is under the configured
timeout, so nudgeDeadline resets it each time. The request completes in
~3.08s -- 10.3x the configured timeout -- with no error, no retry, no
--low-level-retries engagement:

=== RUN   TestTimeoutDoesNotBoundSlowTrickle
    stall_repro_test.go:99: configured --timeout=300ms; request actually took
    3.082205503s (10.3x over) and SUCCEEDED with no error, no retry, and no
    --low-level-retries engagement -- --timeout does not bound a slow trickle
--- PASS: TestTimeoutDoesNotBoundSlowTrickle (3.08s)

(Test is deliberately a require-based assertion of the current, buggy
behavior succeeding, i.e. it currently passes -- happy to invert it to fail
against current behavior if that's more useful for triage.)

What is your rclone version (output from rclone version)

rclone v1.74.4
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-137-generic (x86_64)
- os/type: linux
- os/arch: amd64

(Confirmed the same mechanism is present on current master, see repro branch above.)

Which cloud storage system are you using?

Amazon S3 (provider: DigitalOcean, Spaces)

The command you were trying to run

rclone copy /backup/backup/db/ do-sfo:corvex-backups-sfo3-cold/prod/ \
  --transfers=8 --checkers=8 --retries=3 --retries-sleep=60s \
  --stats-one-line --stats=5m --size-only --s3-no-check-bucket \
  --fast-list --tpslimit=10 --tpslimit-burst=20

Why this matters / suggested direction

--timeout's doc string and most users' mental model of it is "kill a stalled
transfer" -- but as implemented it only detects a fully silent connection,
not a slow-but-technically-alive one. In practice this means a wedged
multipart upload (e.g. an orphaned/desynced multipart session on the S3 side,
or a misbehaving proxy) can run forever with zero visibility, since nothing
in rclone's error/retry machinery ever gets a chance to fire.

A minimum-throughput / stall-detection guard (in the spirit of curl's
--speed-limit/--speed-time) on top of the existing idle-timeout would
close this: track bytes moved per timeout window and error out (surfacing to
the normal --retries path) if that falls under some threshold, rather than
only checking "was anything read/written at all."

Happy to work up a PR for this (I already have the repro and read through
fs/fshttp and the S3 backend's HTTP client wiring to trace the mechanism),
but wanted to raise it here first in case there's already a preferred design,
a reason this hasn't been done, or a "yes but do X instead" before I sink time
into an implementation. Let me know what shape of fix (if any) you'd want to
see, and I'll send a PR.

Start with fs/fshttp/dialer.go, especially timeoutConn.nudgeDeadline, and inspect fs/fshttp/stall_repro_test.go and TestTimeoutDoesNotBoundSlowTrickle. Determine how a slow trickle should be detected while preserving the existing idle deadline, then verify that the resulting error reaches the normal retry path and that the repro no longer completes indefinitely without an error.