oxidecomputer / oxidecomputer/omicron

instance stuck in starting, propolis zone fails to come up

Open
#7,257 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

On dogfood (rack2) after a mupdate to https://github.com/oxidecomputer/omicron/commit/020fde11e74b09202d6c53b491768bc4912903af
I have an instance that is stuck in starting.

The propolis zone responsible is on sled 23 (BRM42220016).
I see this zone has either started and failed in some way, or has never fully started:

BRM42220016 # zlogin oxz_propolis-server_f4019957-5d9d-45a0-97d8-e2c9c523a3e7
[Connected to zone 'oxz_propolis-server_f4019957-5d9d-45a0-97d8-e2c9c523a3e7' pts/3]
Last login: Sun Dec 15 02:50:00 on pts/3
The illumos Project     helios-2.0.23049        December 2024
root@oxz_propolis:~# svcs -x
svcs: Could not bind to repository server: repository server unavailable.  Exiting.
root@oxz_propolis:~# ps -ef 
     UID   PID  PPID   C    STIME TTY         TIME CMD
  netadm  9033  8945   0 01:15:01 ?           0:00 /lib/inet/ipmgmtd
    root  8962  8945   0 01:14:59 ?           0:00 /sbin/init
    root  9082  9081   0        - ?           0:00 <defunct>
    root  9237  9236   0        - ?           0:00 <defunct>
    root  8945  8945   0 01:14:59 ?           0:00 zsched
    root  9236  9079   0 01:15:27 ?           0:00 sulogin
    root  9079  8945   0 01:15:06 ?           0:00 /lib/svc/bin/svc.startd
    root  9081  9079   0 01:15:06 console     0:00 sulogin
  netcfg  9026  8945   0 01:15:01 ?           0:00 /lib/inet/netcfgd
    root 15039 15014   0 16:23:34 pts/3       0:00 ps -ef
    root 15013  8945   0 16:23:26 pts/3       0:00 /usr/bin/login -z global -f root
    root 15014 15013   0 16:23:26 pts/3       0:00 -bash

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dogfood rack2 report, the mupdate commit, and the propolis zone on sled 23 (BRM42220016). Reproduce or inspect the failure using svcs -x and ps -ef inside the zone, then trace why the service repository is unavailable. Done means identifying the startup failure and verifying that the affected instance no longer remains stuck in starting.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.