oxidecomputer / oxidecomputer/omicron
RSS of single-node non-gimlet deployment got stuck trying to time sync
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I tried to deploy current main on dunkin, a helios lab machine (dfb6853f).
At first, I hit issues due to chrony not being installed (#4094), but I installed that package, then restarted the baseline service.
After a few laps with trying to stand up omicron (including running an uninstall command from omicron-package and running the destroy virtual hw script between laps), sled agent is stuck trying to timesync.
These are the zones that came up:
global
sidecar_softnpu
oxz_switch
oxz_internal_dns_f8491614-6fd1-40f5-88c7-6d433054ec43
oxz_internal_dns_9295a7fc-6942-4ee6-98dc-803429a63d98
oxz_internal_dns_7e1a3d47-8086-45a6-b88d-3700f8b9cff7
oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab
The ntp zone thinks the clock is well synced:
root@oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab:~# dladm
LINK CLASS MTU STATE BRIDGE OVER
vopte0 vnic 1500 up -- ?
oxControlService4 vnic 9000 up -- ?
root@oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab:~# chronyc tracking
Reference ID : 7F7F0101 ()
Stratum : 10
Ref time (UTC) : Mon Sep 25 15:05:11 2023
System time : 0.000000000 seconds fast of NTP time
Last offset : +0.000000000 seconds
RMS offset : 0.000000000 seconds
Frequency : 0.117 ppm fast
Residual freq : +0.000 ppm
Skew : 0.000 ppm
Root delay : 0.000000000 seconds
Root dispersion : 0.000000000 seconds
Update interval : 0.0 seconds
Leap status : Normal
On the sled agent side, I see:
15:05:12.485Z INFO SledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, r
ef_time: 1695654311.4054208, correction: 0.0 }
file = sled-agent/src/rack_setup/service.rs:476
15:05:12.485Z WARN SledAgent (RSS): Time is not yet synchronized
error = "Time is synchronized on 0/1 sleds"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at sled-agent/src/rack_setup/service.rs:476 and compare the sled agent's Timesync result with the chronyc tracking output from the NTP zone. Reproduce the single-node RSS deployment on dunkin if available; done means the sled agent no longer remains stuck reporting that time is not synchronized.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100