oxidecomputer / oxidecomputer/omicron
wicket preflight check can't undo a failed setup
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
During a failed preflight check on r12 release (see https://github.com/oxidecomputer/omicron/issues/7459) I was left with a qsfp port still partially configured.
The config.toml in this case had incomplete info for switch 0 qsfp0:
[rack_network_config.switch0.qsfp0]
routes = [ ]
addresses = [ ]
uplink_port_speed = "40G"
uplink_port_fec = "none"
autoneg = true
bgp_peers = []
The preflight check tried to setup qsfp0, failed, then could not tear down what it had created.
alan@jeeves $ ssh -t wicket@fe80::aa40:25ff:fe05:2702%london_sw0tp0 preflight uplink
Warning: Permanently added 'fe80::aa40:25ff:fe05:2702%london_sw0tp0' (ED25519) to the list of known hosts.
✔ Configuring switch (11.398809ms)
configured Qsfp(Qsfp("qsfp0"))/0: ips [], routes []
✔ Waiting for L1 link up (1.004374661s)
missing metadata
✔ Configuring IP address in host OS (37.391µs)
configured uplinks/qsfp0_0
✔ Configuring routing in host OS (35.097µs)
added routes []
Checking for external DNS connectivity (60.121132165s)
DNS server 1.1.1.1 A query attempt 1: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
...
DNS server 1.1.1.1 A query attempt 31: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
DNS server 9.9.9.9 A query attempt 1: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
...
DNS server 9.9.9.9 A query attempt 31: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
* Checking for external NTP connectivity (6.051µs)
missing metadata
no NTP IP addresses known (DNS failure?)
✔ Cleaning up host OS routing configuration (56.628µs)
missing metadata
✘ Cleaning up host OS IP address configuration (20.848173ms)
missing metadata
failed to remove uplink SMF property "uplinks/qsfp0_0": command failed (status exit status: 1); stdout="" stderr="svccfg: No such property uplinks/qsfp0_0.\n"
Connection to fe80::aa40:25ff:fe05:2702%london_sw0tp0 closed.
Afterwards, I still had a config on qsfp0 on switch 0:
alan@jeeves $ ssh -t wicket@fe80::aa40:25ff:fe05:2702%london_sw0tp0 preflight uplink
Warning: Permanently added 'fe80::aa40:25ff:fe05:2702%london_sw0tp0' (ED25519) to the list of known hosts.
✘ Configuring switch (522.656µs)
missing metadata
uplink already configured - is rack already initialized?
Connection to fe80::aa40:25ff:fe05:2702%london_sw0tp0 closed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the preflight uplink flow and reproduce the incomplete qsfp0 configuration described here, using issue 7459 for the failed setup context. Trace the cleanup after the DNS failure; done means a failed preflight removes any partial qsfp0 configuration so a subsequent run does not report that the uplink is already configured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100