oxidecomputer / oxidecomputer/omicron
`dpd` can't find out which switch it's connected to via MGS
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
I deployed 6c3b861 on dublin, and both switch zone shell prompts fail to show which switch we're connected to:
root@oxz_switchnull:~#
There are no services in maintenance and the rack appears to be generally healthy. I can do normal debugging operations, the console is accessible, etc.
The MGS logs show errors due to a client from ::1 failing to provide an api-version header; e.g.,
19:37:58.531Z ERRO a69bd92c-8592-4313-8426-a463c70f45e8 (dropshot): failed to determine request API version
error = HttpError { status_code: 400, error_code: None, external_message: "missing expected header \\"api-version\\"", internal_message: "missing expected header \\"api-version\\"", headers: None }
file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.3/src/versioning.rs:67
local_addr = [::1]:12225
method = GET
remote_addr = [::1]:49074
req_id = 46208126-4acf-480a-a64f-93a86c72db78
uri = /local/switch-id
I believe what's going on here is:
- #8917 added API versioning to MGS
progenitorstarted sending theapi-versionheader in version0.10.0, which released in Aprilomicronpulls in a specific dendrite by commit- The dendrite commit we're currently pulling in is from Aug 7 (about 3 weeks ago).
dpdin the dendrite repo usesgateway_clientfromomicron's main branch.- On the dendrite commit we're pulling in from Aug 7, the omicron commit it's using for its
gateway_client(and other crates) is from February (!!) (https://github.com/oxidecomputer/omicron/tree/e2f9776a2f86de2d4cd35926302bb99cedbdb3d4), which predates progenitor 0.10 and explains why we're not getting anapi-versionheader
Some suggestions from discussions in chat:
- We have a reflector bot that's supposed to open PRs in omicron, but we haven't gotten one for dendrite in quite some time. @augustuswm was going to check on this.
- We don't have automation in dendrite to pull in new omicron bits, but probably should.
Short term plan to fix this:
- https://github.com/oxidecomputer/dendrite/pull/112 updates dendrite's dependency to a new omicron.
- Once that lands, we'll need to update omicron to pull in the new dendrite.
I'm not sure how much stuff is broken by this; maybe not a lot? But seeing switchnull usually means something more serious is broken, which is pretty misleading in this case.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with package-manifest.toml and the dendrite dependency revision described in the issue; review dendrite PR #112 and the referenced omicron and Cargo.lock commits. Confirm that the updated dependency sends the api-version header to MGS, then verify that dpd switch-zone shell prompts show the connected switch instead of switchnull.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100