oxidecomputer / oxidecomputer/omicron

ntp zone fails to come up after sled restart

Open
#8,070 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

On Dublin I had rebooted sled 15

When that sled restarted, it failed to bring the ntp zone fully online.

Sled agent reports:

14:17:13.265Z WARN SledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_719d1d2a-487a-4a27-b7a2-1cb212e65f08") failed: Property not found. retry in 767.360061ms
    file = illumos-utils/src/svc.rs:36
    zone = oxz_ntp_719d1d2a-487a-4a27-b7a2-1cb212e65f08
14:17:14.050Z WARN SledAgent (ServiceManager): wait for service svc:/milestone/single-user:default in zone Some("oxz_ntp_719d1d2a-487a-4a27-b7a2-1cb212e65f08") failed: Property not found. retry in 1.288516879s
    file = illumos-utils/src/svc.rs:36
    zone = oxz_ntp_719d1d2a-487a-4a27-b7a2-1cb212e65f08

NTP zone reports:

root@oxz_ntp_719d1d2a:~# svcs -x 
svc:/network/ip-interface-management:default (IP interface management)
 State: maintenance since Sun Dec 28 00:05:05 1986
Reason: Start method died on Killed (9).
   See: http://illumos.org/msg/SMF-8000-KS
   See: ipadm(8)
   See: /var/svc/log/network-ip-interface-management:default.log
Impact: 24 dependent services are not running.  (Use -v for list.)

svc:/network/physical:nwam (physical network interface autoconfiguration)
 State: disabled since Sun Dec 28 00:04:05 1986
Reason: Disabled by an administrator.
   See: http://illumos.org/msg/SMF-8000-05
   See: nwamd(8)
   See: http://hub.opensolaris.org/bin/view/Project+nwam/
Impact: 20 dependent services are not running.  (Use -v for list.)

svc:/network/routing/ndp:default (in.ndpd IPv6 neighbour discovery daemon)
 State: disabled since Sun Dec 28 00:05:34 1986
Reason: Disabled by an administrator.
   See: http://illumos.org/msg/SMF-8000-05
   See: in.ndpd(8)
Impact: 2 dependent services are not running.  (Use -v for list.)

Then, the service itself tells us this:

root@oxz_ntp_719d1d2a:~# 
root@oxz_ntp_719d1d2a:~# cat $(svcs -L ip-interface-management) | looker
[ Dec 28 00:04:05 Enabled. ]
[ Dec 28 00:04:05 Executing start method ("/lib/svc/method/net-ipmgmt"). ]
[ Dec 28 00:05:05 Method or service exit timed out.  Killing contract 159. ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with illumos-utils/src/svc.rs:36 and the SledAgent ServiceManager path that waits for the NTP zone milestone. Review the reported svc:/network/ip-interface-management failure and its /var/svc/log/network-ip-interface-management:default.log output. Done means the NTP zone reaches single-user online after a sled restart without the network service timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.