propolis zone never got to networking?

this is both collecting @paudmir's observations from #7257 into a distinct issue, and some notes I've taken as I'm looking at it a bit more. it seems like the problems are different even though the outcomes are similar. Paulina's notes from there:

We had a bunch of crucible-agent cores piling up and services down:

Services that are down:
BRM13250012

 3  BRM13250012        ok: svc:/network/ip-interface-management:default (IP interface management)
  Zone: oxz_propolis-server_ffe97f34-1655-4393-a250-03d67a80cbf1
 State: maintenance since Thu Sep  3 18:21:15 2026
Reason: Start method died on Killed (9).
   See: http://illumos.org/msg/SMF-8000-KS
   See: ipadm(8)
   See: /pool/ext/5c1cf4d3-af0c-48c2-92d0-1308a8dd9787/crypt/zone/oxz_propolis-server_ffe97f34-1655-4393-a250-03d67a80cbf1/root/var/svc/log/network-ip-interface-management:default.log
Impact: 21 dependent services are not running.  (Use -v for list.)

svc:/network/physical:nwam (physical network interface autoconfiguration)
  Zone: oxz_propolis-server_ffe97f34-1655-4393-a250-03d67a80cbf1
 State: disabled since Thu Sep  3 18:20:15 2026
Reason: Disabled by an administrator.
   See: http://illumos.org/msg/SMF-8000-05
   See: nwamd(8)
   See: http://hub.opensolaris.org/bin/view/Project+nwam/
Impact: 17 dependent services are not running.  (Use -v for list.)

root@oxz_propolis:/var/svc/log# ps -ef
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root  8597  8596   0        - ?           0:00 <defunct>
    root  8559  7897   0   Sep 03 ?           0:03 /usr/sbin/nscd
    root  7897  7897   0   Sep 03 ?           0:00 zsched
    root  7913  7897   0   Sep 03 ?           0:00 /sbin/init
    root  8596  7920   0   Sep 03 console     0:00 sulogin
    root 26617 20818   0 18:46:05 pts/5       0:00 ps -ef
    root  8407  7897   0   Sep 03 ?           0:00 /usr/lib/pfexecd
    root 20817  7897   0 18:44:24 pts/5       0:00 /usr/bin/login -z global -f root
    root  7922  7897   0   Sep 03 ?           3:45 /lib/svc/bin/svc.configd
    root 20818 20817   0 18:44:24 pts/5       0:00 -bash
    root  7920  7897   0   Sep 03 ?           0:02 /lib/svc/bin/svc.startd
    root  8341  7897   0   Sep 03 ?           0:00 /usr/lib/fm/fmd/fmd
  netcfg  7968  7897   0   Sep 03 ?           0:00 /lib/inet/netcfgd

Instance in STARTING:

== INSTANCE ====================================================================
                        ID: 852ebb78-dbe6-404f-966e-cb9a566fa8ba
                project ID: 7adf3d37-e230-47fd-b0cc-8ced0c51e4d5
                      name: startstop-alpine-4c-16g
               description: stop/start loop test instance
                created at: 2026-09-03 05:32:05.947838 UTC
          last modified at: 2026-09-03 19:45:59.583164 UTC

== VMM =========================================================================
                                  ID: ffe97f34-1655-4393-a250-03d67a80cbf1
                         instance ID: 852ebb78-dbe6-404f-966e-cb9a566fa8ba
                          created at: 2026-09-03 18:16:41.451719 UTC
                               state: starting
                          updated at: 2026-09-03T18:16:42.637007Z (generation 2)
                    propolis address: fd00:1122:3344:129::1:1d2a:12400
                             sled ID: 781ab3c9-ff3c-428a-9f3e-895a1e86a111
                         sled serial: BRM13250012
                        CPU platform: AmdMilan

some more notes as I've looked at the zone today.. svcs in the zone is

root@oxz_propolis:~# svcs
STATE          STIME    FMRI
online         Sep_03   svc:/system/svc/restarter:default
online         Sep_03   svc:/system/early-manifest-import:default
online         Sep_03   svc:/network/netcfg:default
online         Sep_03   svc:/network/datalink-management:default
online         Sep_03   svc:/system/filesystem/root:default
online         Sep_03   svc:/system/boot-archive:default
online         Sep_03   svc:/system/filesystem/usr:default
online         Sep_03   svc:/system/device/local:default
online         Sep_03   svc:/network/initial:default
online         Sep_03   svc:/milestone/devices:default
online         Sep_03   svc:/network/ipsec/ipsecalgs:default
online         Sep_03   svc:/system/device/audio:default
online         Sep_03   svc:/system/filesystem/minimal:default
online         Sep_03   svc:/network/ipsec/policy:default
online         Sep_03   svc:/system/rmtmpfiles:default
online         Sep_03   svc:/network/netmask:default
online         Sep_03   svc:/system/fmd:default
online         Sep_03   svc:/system/auditset:default
online         Sep_03   svc:/system/pfexec:default
online         Sep_03   svc:/system/hostid:default
online         Sep_03   svc:/system/logadm-upgrade:default
online         Sep_03   svc:/system/pkgserv:default
online         Sep_03   svc:/system/rbac:default
online         Sep_03   svc:/network/service:default
online         Sep_03   svc:/system/cryptosvc:default
online         Sep_03   svc:/system/keymap:default
online         Sep_03   svc:/system/name-service-cache:default
online         Sep_03   svc:/milestone/name-services:default
online         Sep_03   svc:/system/manifest-import:default
online         Sep_03   svc:/system/coreadm:default
online         Sep_03   svc:/system/boot-config:default
online         Sep_03   svc:/network/routing-setup:default
offline        Sep_03   svc:/network/loopback:default
offline        Sep_03   svc:/network/physical:default
offline        Sep_03   svc:/system/identity:node
offline        Sep_03   svc:/milestone/network:default
offline        Sep_03   svc:/milestone/single-user:default
offline        Sep_03   svc:/milestone/sysconfig:default
offline        Sep_03   svc:/system/filesystem/local:default
offline        Sep_03   svc:/system/system-log:default
offline        Sep_03   svc:/network/inetd-upgrade:default
offline        Sep_03   svc:/system/identity:domain
offline        Sep_03   svc:/system/utmp:default
offline        Sep_03   svc:/system/cron:default
offline        Sep_03   svc:/milestone/multi-user:default
offline        Sep_03   svc:/network/ssh:default
offline        Sep_03   svc:/milestone/multi-user-server:default
offline        Sep_03   svc:/network/shares/group:default
offline        Sep_03   svc:/network/iptun:default
offline        Sep_03   svc:/system/console-login:default
offline        Sep_03   svc:/system/boot-archive-update:default
offline        Sep_03   svc:/system/update-man-index:default
offline        Sep_03   svc:/system/illumos/propolis-server:default
maintenance    Sep_03   svc:/network/ip-interface-management:default

and, as I wasn't sure what all depends on ip-interface-management (though the name certainly says what it is):

root@oxz_propolis:~# svcs -D svc:/network/ip-interface-management:default
STATE          STIME    FMRI
disabled       Sep_03   svc:/network/physical:nwam
disabled       Sep_03   svc:/network/install:default
offline        Sep_03   svc:/network/loopback:default
offline        Sep_03   svc:/network/physical:default
offline        Sep_03   svc:/network/iptun:default

I wasn't sure about what the disabled services are about but that seems normal comparing to another Propolis zone elsewhere:

root@oxz_propolis:~# svcs -D svc:/network/ip-interface-management:default
STATE          STIME    FMRI
disabled       Sep_02   svc:/network/physical:nwam
disabled       Sep_02   svc:/network/install:default
online         Sep_02   svc:/network/loopback:default
online         Sep_02   svc:/network/physical:default
online         Sep_02   svc:/network/iptun:default

so.. setting aside the disabled services, the zone never got to /network/physical or /network/loopback, and on the other side Propolis does expect there to be networking:

root@oxz_propolis:~# svcs -d svc:/system/illumos/propolis-server:default
STATE          STIME    FMRI
offline        Sep_03   svc:/milestone/network:default
offline        Sep_03   svc:/milestone/multi-user:default

so this is all consistent with "we didn't even get to starting propolis-server". which is an unfortunate finding because if we look at what /network/ip-interface-management did do, it's ...

root@oxz_propolis:~# cat /var/svc/log/network-ip-interface-management:default.log
[ Sep  3 18:20:15 Enabled. ]
[ Sep  3 18:20:15 Executing start method ("/lib/svc/method/net-ipmgmt"). ]
[ Sep  3 18:21:15 Method or service exit timed out.  Killing contract 7480. ]

which isn't much to work with. /lib/svc/method/net-ipmgmt here is this Bash:

root@oxz_propolis:/var# cat /lib/svc/method/net-ipmgmt
#!/sbin/sh
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License (the "License").
# You may not use this file except in compliance with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# CDDL HEADER END
#
#
# Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
# Copyright (c) 2012, Joyent, Inc. All rights reserved.
#
# This daemon stores address object to logical interface number mappings
# (among other things) and reads/writes from/to ipmgmtd data store.
#

. /lib/svc/share/smf_include.sh

if [ -z "$SMF_FMRI" ]; then
        echo "this script can only be invoked by smf(7)"
        exit $SMF_EXIT_ERR_NOSMF
fi

#
# network/ip-interface-management:default service is always enabled by default.
# When the non-global shared-IP stack zone boots, it tries to bring up this
# service as well. If we don't start a background process and simply exit the
# service, the service will go into maintenance mode and so will all it's
# dependents.  Ideally we would simply exit with SMF_EXIT_NODAEMON, but since
# this method is also used in an S10C zone, where support for SMF_EXIT_NODAEMON
# does not exist, we have to stick around.
#
# In S10C zone (where this script is also used) smf_isnonglobalzone
# function is unavailable in smf_include.sh
#
if [ `/sbin/zonename` != global ]; then
        if [ `/sbin/zonename -t` = shared ]; then
                (while true ; do sleep 3600 ; done) &
                exit $SMF_EXIT_OK
        fi
fi

#
# We must be now in a global zone or non-global zone with exclusive-IP stack.
# Start the ipmgmtd daemon.
#
if /lib/inet/ipmgmtd ; then
        exit $SMF_EXIT_OK
else
        exit $SMF_EXIT_ERR_FATAL
fi

assuming we did get to running the shell, we would have discovered that

root@oxz_propolis:~# /sbin/zonename
oxz_propolis-server_ffe97f34-1655-4393-a250-03d67a80cbf1
root@oxz_propolis:~# /sbin/zonename -t
exclusive

and pretty much immediately gotten to backgrounding a while true and said OK?

I'm not fully satisfied by that explanation, because in a more normal zone, we see /lib/inet/ipmgmtd:

root@oxz_propolis:~# ps -ef | grep ipmgm
  netadm 16821 16724   0   Sep 02 ?           0:03 /lib/inet/ipmgmtd
    root 21509  3029   0 20:45:25 pts/7       0:00 grep ipmgm

where there is not one here:

root@oxz_propolis:~# ps -ef | grep ipmgm
    root  6959  6429   0 20:45:44 pts/3       0:00 grep ipmgm

but I'm... not sure what would have run that in the healthy case. grep -R ipmgmt /var has similarly not turned up anything, so I'm not sure how to debug what net-ipmgmt didn't do... or did, and failed to?

Start with /lib/svc/method/net-ipmgmt and /var/svc/log/network-ip-interface-management:default.log, then compare the failed zone with a healthy Propolis zone. Trace why /lib/inet/ipmgmtd is absent and why /network/physical and /network/loopback remain offline. Done means the zone reaches networking and svc:/system/illumos/propolis-server:default can start.