benbjohnson / benbjohnson/litestream
Docs: add a troubleshooting note for the "counter cannot decrease in value" panic (unstable clocksource)
- Dominant language
- Go
- Stars
- 14.4k
- Forks
- 414
- Avg merge
- 7d 1h
- Merged PRs (30d)
- 21
Description
Companion to #1488, which proposes clamping the delta at `db.go:1276`. This one is about the docs, because even once the panic is fixed the underlying condition is worth telling operators about — and if the code fix lands, the symptom disappears while the broken clock remains.
## Suggested addition
A short troubleshooting entry, something like:
> ### `panic: counter cannot decrease in value`
>
> litestream measures elapsed time with `time.Since` and reports it as a Prometheus counter. If the host's `CLOCK_MONOTONIC` runs backwards, that value is negative and the counter panics, killing the process — typically in a `Restart=always` loop, so replication stops while the unit still looks like it is running.
>
> This is a host clock problem, not a litestream configuration problem. Check the clocksource:
>
> ```
> cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> ```
>
> On a KVM/QEMU guest showing `tsc`, switch to the paravirtualised clock:
>
> ```
> echo kvm-clock | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
> ```
>
> Make it persistent — a sysfs write does not survive reboot. Unsynchronised TSC across vCPUs is the usual cause.
## Why it is worth a doc entry
The message names a metrics counter and the stack trace names `syncLocked`. Neither points at the clock. In our case we eliminated concurrent writers, duplicate replication processes, credentials and config first — all fine, and uploads were succeeding right up to each panic. Reading `db.go:1276` was what identified it.
Measured on the affected host:
| clocksource | samples (45 s) | backwards jumps | magnitude |
|---|---|---|---|
| `tsc` | 75,200,443 | **4** | 0.27 – 0.79 ms |
| `kvm-clock` | 69,525,047 | **0** | — |
13 panics in ~15 minutes on `tsc`; 0 panics and 0 restarts after switching, with uploads continuing normally.
## One caution on the "already fixed" reading
Clamping in #1488 is right, but it removes the *symptom*. A host whose monotonic clock regresses will still mis-measure every interval litestream uses — sync durations, retry backoff, any timeout. litestream happens to be the process that made it visible because it treats a negative interval as fatal; on our box roughly twenty other processes absorbed the same readings without a word. A note telling operators to check the clocksource keeps that diagnostic value after the panic is gone.
Happy to open a PR against the docs if the wording above is close to what you'd want.
Contributor guide
Research direction
Start in the repository's documentation entry points and read db.go:1276 for the panic context. Add a concise troubleshooting entry covering the clocksource checks, KVM/QEMU guidance, and persistence warning; done means operators can identify the host clock problem and follow the documented commands.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux, prometheus
- Domain
- documentation, observability, operating-systems
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100