ChainSafe / ChainSafe/lodestar

Detect and warn about clock drift

Open
#6,037 0 comments 0 reactions 0 assignees View on GitHub
meta-discussion meta-feature-request
Dominant language
TypeScript
Stars
1.4k
Forks
483
Avg merge
1d 16h
Merged PRs (30d)
150

Description

### Problem description

There have been a lot of issues reported by users due to their system clock being skewed / not synced correctly. Those range from client not being able to sync, to missed head votes / missed attestations, or just in general degraded effectiveness without logging any errors or warnings.

Determining that a clock drift as the root cause can be pretty difficult and most of the time just comes down to asking the user to check their system time via terminal or clock drift metric.

Fixing the issue itself it pretty easy by installing [chrony / ntp for time synchronization](https://eth-docker.net/Usage/LinuxSecurity/#time-synchronization-on-linux).

### Solution description

Lodestar could try to detect a clock drift by observing messages from the network and if there are many messages that come too early / late a warning could be logged to inform the user. This solution relies on heuristics and might not be that simple to implement but would most likely be the best to solve issues mentioned above.

### Alternative solution

Another more simple option is to just detect a time discrepancy between the beacon node and validator client. This could be done by comparing the clock slot of the validator client with the head slot + sync distance reported by the beacon node sync API. The problem with this solution is that it might not detect all clock drift issues as the granularity is slot based and it does not help to detect a clock drift if both instances run on the same server which is the setup for most solo stakers. This solution has been [implemented by Lighthouse](https://github.com/sigp/lighthouse/blob/c3321dddb7e8d8d55a39dadaa6be4abee05dd9a4/validator_client/src/check_synced.rs#L62-L74) already but might [produce false positives](https://github.com/ChainSafe/lodestar/issues/5555) due to a different interpretation by clients of what value should be used for `sync_distance` if the node is synced https://github.com/sigp/lighthouse/issues/3421#issuecomment-1764281107.

### Additional context

At the moment, Lodestar would throw an error if the clock drift is significant enough to trigger a `ATTESTATION_ERROR_FUTURE_SLOT` / `BLOCK_ERROR_FUTURE_SLOT` error but that has only been observed once (or twice) so far.

There is also [metric to track clock drift](https://grafana-lodestar.chainsafe.io/d/lodestar_summary/lodestar?orgId=1&refresh=10s&viewPanel=102) on the Lodestar summary dashboard but this requires to have metrics enabled and it is not something a user would actively look at during normal operation whereas a warning log would most likely get their attention.

![image](https://github.com/ChainSafe/lodestar/assets/38436224/7ecd81ff-9173-447c-9331-c15ebd0f9067)

### Related
- https://github.com/ChainSafe/lodestar-ansible-development/issues/197
- https://github.com/dappnode/DNP_CORE/pull/145
- [Discussion in Eth R&D discord](https://discord.com/channels/595666850260713488/598292067260825641/1161251107352367125)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.