benbjohnson / benbjohnson/litestream

Docs: add a troubleshooting note for the "counter cannot decrease in value" panic (unstable clocksource)

Open Beginner friendly
#1,489 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14.4k
Forks
414
Avg merge
7d 1h
Merged PRs (30d)
21

Description

Companion to #1488, which proposes clamping the delta at `db.go:1276`. This one is about the docs, because even once the panic is fixed the underlying condition is worth telling operators about — and if the code fix lands, the symptom disappears while the broken clock remains.

## Suggested addition

A short troubleshooting entry, something like:

> ### `panic: counter cannot decrease in value`
>
> litestream measures elapsed time with `time.Since` and reports it as a Prometheus counter. If the host's `CLOCK_MONOTONIC` runs backwards, that value is negative and the counter panics, killing the process — typically in a `Restart=always` loop, so replication stops while the unit still looks like it is running.
>
> This is a host clock problem, not a litestream configuration problem. Check the clocksource:
>
> ```
> cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> ```
>
> On a KVM/QEMU guest showing `tsc`, switch to the paravirtualised clock:
>
> ```
> echo kvm-clock | sudo tee /sys/devices/system/clocksource/clocksource0/current_clocksource
> ```
>
> Make it persistent — a sysfs write does not survive reboot. Unsynchronised TSC across vCPUs is the usual cause.

## Why it is worth a doc entry

The message names a metrics counter and the stack trace names `syncLocked`. Neither points at the clock. In our case we eliminated concurrent writers, duplicate replication processes, credentials and config first — all fine, and uploads were succeeding right up to each panic. Reading `db.go:1276` was what identified it.

Measured on the affected host:

| clocksource | samples (45 s) | backwards jumps | magnitude |
|---|---|---|---|
| `tsc` | 75,200,443 | **4** | 0.27 – 0.79 ms |
| `kvm-clock` | 69,525,047 | **0** | — |

13 panics in ~15 minutes on `tsc`; 0 panics and 0 restarts after switching, with uploads continuing normally.

## One caution on the "already fixed" reading

Clamping in #1488 is right, but it removes the *symptom*. A host whose monotonic clock regresses will still mis-measure every interval litestream uses — sync durations, retry backoff, any timeout. litestream happens to be the process that made it visible because it treats a negative interval as fatal; on our box roughly twenty other processes absorbed the same readings without a word. A note telling operators to check the clocksource keeps that diagnostic value after the panic is gone.

Happy to open a PR against the docs if the wording above is close to what you'd want.

Contributor guide

Open the contributing guide

Research direction

Start in the repository's documentation entry points and read db.go:1276 for the panic context. Add a concise troubleshooting entry covering the clocksource checks, KVM/QEMU guidance, and persistence warning; done means operators can identify the host clock problem and follow the documented commands.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux, prometheus
Domain
documentation, observability, operating-systems
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.