oxidecomputer / oxidecomputer/omicron
Services going into maintenance mode within zones cause rack setup to hang.
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I'm working on launching a 4-node multi-switch/multi-node rack as a topology of virtual machines. On my last run rack setup got stuck. The issue appears to be an SMF service within a zone in maintenance mode.
root@g0:/opt/cargo-bay/omicron# svcs -xvZ
svc:/network/ip-interface-management:default (IP interface management)
Zone: oxz_cockroachdb_351a3bde-9b76-4b4c-80e8-f8b2d63702da
State: maintenance since Tue Oct 31 15:27:25 2023
Reason: Start method died on Killed (9).
See: http://illumos.org/msg/SMF-8000-KS
See: man -M /usr/share/man -s 8 ipadm
See: /pool/ext/14b4dc84-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_351a3bde-9b76-4b4c-80e8-f8b2d63702da/root/var/svc/log/network-ip-interface-management:default.log
Impact: 21 dependent services are not running:
svc:/network/loopback:default
svc:/system/identity:node
svc:/milestone/single-user:default
svc:/milestone/sysconfig:default
svc:/milestone/multi-user:default
svc:/oxide/cockroachdb:default
svc:/milestone/multi-user-server:default
svc:/system/system-log:default
svc:/system/utmp:default
svc:/system/console-login:default
svc:/network/ssh:default
svc:/system/filesystem/local:default
svc:/system/cron:default
svc:/system/update-man-index:default
svc:/network/inetd-upgrade:default
svc:/system/boot-archive-update:default
svc:/network/shares/group:default
svc:/network/physical:default
svc:/milestone/network:default
svc:/system/identity:domain
svc:/network/iptun:default
svc:/network/physical:nwam (physical network interface autoconfiguration)
Zone: oxz_cockroachdb_351a3bde-9b76-4b4c-80e8-f8b2d63702da
State: disabled since Tue Oct 31 15:26:24 2023
Reason: Disabled by an administrator.
See: http://illumos.org/msg/SMF-8000-05
See: man -M /usr/share/man -s 8 nwamd
See: http://hub.opensolaris.org/bin/view/Project+nwam/
Impact: 17 dependent services are not running:
svc:/milestone/network:default
svc:/milestone/single-user:default
svc:/milestone/sysconfig:default
svc:/milestone/multi-user:default
svc:/oxide/cockroachdb:default
svc:/milestone/multi-user-server:default
svc:/system/system-log:default
svc:/system/utmp:default
svc:/system/console-login:default
svc:/network/ssh:default
svc:/system/filesystem/local:default
svc:/system/cron:default
svc:/system/update-man-index:default
svc:/network/inetd-upgrade:default
svc:/system/boot-archive-update:default
svc:/network/shares/group:default
svc:/network/iptun:default
svc:/network/ip-interface-management:default (IP interface management)
Zone: oxz_cockroachdb_8a0fd8a9-c1ea-49ca-bb80-67d5d2a5d4dc
State: maintenance since Tue Oct 31 15:27:25 2023
Reason: Start method died on Killed (9).
See: http://illumos.org/msg/SMF-8000-KS
See: man -M /usr/share/man -s 8 ipadm
See: /pool/ext/f4b4dc84-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8a0fd8a9-c1ea-49ca-bb80-67d5d2a5d4dc/root/var/svc/log/network-ip-interface-management:default.log
Impact: 21 dependent services are not running:
svc:/network/loopback:default
svc:/system/identity:node
svc:/milestone/single-user:default
svc:/milestone/sysconfig:default
svc:/milestone/multi-user:default
svc:/oxide/cockroachdb:default
svc:/milestone/multi-user-server:default
svc:/system/system-log:default
svc:/system/utmp:default
svc:/system/console-login:default
svc:/network/ssh:default
svc:/system/filesystem/local:default
svc:/system/cron:default
svc:/system/update-man-index:default
svc:/network/inetd-upgrade:default
svc:/system/boot-archive-update:default
svc:/network/shares/group:default
svc:/network/physical:default
svc:/milestone/network:default
svc:/system/identity:domain
svc:/network/iptun:default
svc:/network/physical:nwam (physical network interface autoconfiguration)
Zone: oxz_cockroachdb_8a0fd8a9-c1ea-49ca-bb80-67d5d2a5d4dc
State: disabled since Tue Oct 31 15:26:24 2023
Reason: Disabled by an administrator.
See: http://illumos.org/msg/SMF-8000-05
See: man -M /usr/share/man -s 8 nwamd
See: http://hub.opensolaris.org/bin/view/Project+nwam/
Impact: 17 dependent services are not running:
svc:/milestone/network:default
svc:/milestone/single-user:default
svc:/milestone/sysconfig:default
svc:/milestone/multi-user:default
svc:/oxide/cockroachdb:default
svc:/milestone/multi-user-server:default
svc:/system/system-log:default
svc:/system/utmp:default
svc:/system/console-login:default
svc:/network/ssh:default
svc:/system/filesystem/local:default
svc:/system/cron:default
svc:/system/update-man-index:default
svc:/network/inetd-upgrade:default
svc:/system/boot-archive-update:default
svc:/network/shares/group:default
svc:/network/iptun:default
root@g0:/opt/cargo-bay/omicron#
This is not specific to the cockroach zone. I've observed this in other zones such as nexus that also halt sled agent progress.
A gcore of sled agent when the above happened is at catacomb:/data/staff/ry/omicron-in-falcon
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the rack setup path involving sled agent progress and the svcs -xvZ output for zones such as cockroachdb and nexus. Inspect the provided sled-agent gcore at catacomb:/data/staff/ry/omicron-in-falcon and reproduce the multi-switch, multi-node virtual-machine rack setup. Done should mean a service entering maintenance mode no longer leaves rack setup or sled agent progress hanging.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100