oxidecomputer / oxidecomputer/omicron

wicket preflight check can't undo a failed setup

Open
#7,460 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

During a failed preflight check on r12 release (see https://github.com/oxidecomputer/omicron/issues/7459) I was left with a qsfp port still partially configured.

The config.toml in this case had incomplete info for switch 0 qsfp0:

[rack_network_config.switch0.qsfp0]
routes = [ ]
addresses = [ ]
uplink_port_speed = "40G"
uplink_port_fec = "none"
autoneg = true
bgp_peers = []

The preflight check tried to setup qsfp0, failed, then could not tear down what it had created.

alan@jeeves $ ssh -t wicket@fe80::aa40:25ff:fe05:2702%london_sw0tp0 preflight uplink
Warning: Permanently added 'fe80::aa40:25ff:fe05:2702%london_sw0tp0' (ED25519) to the list of known hosts.
✔ Configuring switch (11.398809ms)
    configured Qsfp(Qsfp("qsfp0"))/0: ips [], routes []
✔ Waiting for L1 link up (1.004374661s)
    missing metadata
✔ Configuring IP address in host OS (37.391µs)
    configured uplinks/qsfp0_0
✔ Configuring routing in host OS (35.097µs)
    added routes []
  Checking for external DNS connectivity (60.121132165s)
    DNS server 1.1.1.1 A query attempt 1: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
    ...
    DNS server 1.1.1.1 A query attempt 31: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
    DNS server 9.9.9.9 A query attempt 1: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
    ...
    DNS server 9.9.9.9 A query attempt 31: failed to look up ntp.eng.oxide.computer: proto error: io error: No route to host (os error 148)
* Checking for external NTP connectivity (6.051µs)
    missing metadata
    no NTP IP addresses known (DNS failure?)
✔ Cleaning up host OS routing configuration (56.628µs)
    missing metadata
✘ Cleaning up host OS IP address configuration (20.848173ms)
    missing metadata
    failed to remove uplink SMF property "uplinks/qsfp0_0": command failed (status exit status: 1); stdout="" stderr="svccfg: No such property uplinks/qsfp0_0.\n"
Connection to fe80::aa40:25ff:fe05:2702%london_sw0tp0 closed.

Afterwards, I still had a config on qsfp0 on switch 0:

alan@jeeves $ ssh -t wicket@fe80::aa40:25ff:fe05:2702%london_sw0tp0 preflight uplink
Warning: Permanently added 'fe80::aa40:25ff:fe05:2702%london_sw0tp0' (ED25519) to the list of known hosts.
✘ Configuring switch (522.656µs)
    missing metadata
    uplink already configured - is rack already initialized?
Connection to fe80::aa40:25ff:fe05:2702%london_sw0tp0 closed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the preflight uplink flow and reproduce the incomplete qsfp0 configuration described here, using issue 7459 for the failed setup context. Trace the cleanup after the DNS failure; done means a failed preflight removes any partial qsfp0 configuration so a subsequent run does not report that the uplink is already configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.