oxidecomputer / oxidecomputer/omicron

RSS of single-node non-gimlet deployment got stuck trying to time sync

Open
#4,138 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

development
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I tried to deploy current main on dunkin, a helios lab machine (dfb6853f).

At first, I hit issues due to chrony not being installed (#4094), but I installed that package, then restarted the baseline service.

After a few laps with trying to stand up omicron (including running an uninstall command from omicron-package and running the destroy virtual hw script between laps), sled agent is stuck trying to timesync.

These are the zones that came up:

global
sidecar_softnpu
oxz_switch
oxz_internal_dns_f8491614-6fd1-40f5-88c7-6d433054ec43
oxz_internal_dns_9295a7fc-6942-4ee6-98dc-803429a63d98
oxz_internal_dns_7e1a3d47-8086-45a6-b88d-3700f8b9cff7
oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab

The ntp zone thinks the clock is well synced:

root@oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab:~# dladm
LINK        CLASS     MTU    STATE    BRIDGE     OVER
vopte0      vnic      1500   up       --         ?
oxControlService4 vnic 9000  up       --         ?
root@oxz_ntp_517cf03b-7086-4a1c-b6e7-0054588608ab:~# chronyc tracking
Reference ID    : 7F7F0101 ()
Stratum         : 10
Ref time (UTC)  : Mon Sep 25 15:05:11 2023
System time     : 0.000000000 seconds fast of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 0.117 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 0.000000000 seconds
Root dispersion : 0.000000000 seconds
Update interval : 0.0 seconds
Leap status     : Normal

On the sled agent side, I see:

15:05:12.485Z INFO SledAgent (RSS): Timesync for [fd00:1122:3344:101::1]:12345 TimeSync { sync: false, ref_id: 2139029761, ip_addr: ::, stratum: 10, r
ef_time: 1695654311.4054208, correction: 0.0 }                                                                                                        
    file = sled-agent/src/rack_setup/service.rs:476                                                                                                   
15:05:12.485Z WARN SledAgent (RSS): Time is not yet synchronized           
    error = "Time is synchronized on 0/1 sleds"  

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at sled-agent/src/rack_setup/service.rs:476 and compare the sled agent's Timesync result with the chronyc tracking output from the NTP zone. Reproduce the single-node RSS deployment on dunkin if available; done means the sled agent no longer remains stuck reporting that time is not synchronized.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.