oxidecomputer / oxidecomputer/omicron

`dpd` can't find out which switch it's connected to via MGS

Open
#8,962 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

I deployed 6c3b861 on dublin, and both switch zone shell prompts fail to show which switch we're connected to:

root@oxz_switchnull:~#

There are no services in maintenance and the rack appears to be generally healthy. I can do normal debugging operations, the console is accessible, etc.

The MGS logs show errors due to a client from ::1 failing to provide an api-version header; e.g.,

19:37:58.531Z ERRO a69bd92c-8592-4313-8426-a463c70f45e8 (dropshot): failed to determine request API version
    error = HttpError { status_code: 400, error_code: None, external_message: "missing expected header \\"api-version\\"", internal_message: "missing expected header \\"api-version\\"", headers: None }
    file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.3/src/versioning.rs:67
    local_addr = [::1]:12225
    method = GET
    remote_addr = [::1]:49074
    req_id = 46208126-4acf-480a-a64f-93a86c72db78
    uri = /local/switch-id

I believe what's going on here is:

  • #8917 added API versioning to MGS
  • progenitor started sending the api-version header in version 0.10.0, which released in April
  • omicron pulls in a specific dendrite by commit
  • The dendrite commit we're currently pulling in is from Aug 7 (about 3 weeks ago).
  • dpd in the dendrite repo uses gateway_client from omicron's main branch.
  • On the dendrite commit we're pulling in from Aug 7, the omicron commit it's using for its gateway_client (and other crates) is from February (!!) (https://github.com/oxidecomputer/omicron/tree/e2f9776a2f86de2d4cd35926302bb99cedbdb3d4), which predates progenitor 0.10 and explains why we're not getting an api-version header

Some suggestions from discussions in chat:

  • We have a reflector bot that's supposed to open PRs in omicron, but we haven't gotten one for dendrite in quite some time. @augustuswm was going to check on this.
  • We don't have automation in dendrite to pull in new omicron bits, but probably should.

Short term plan to fix this:

I'm not sure how much stuff is broken by this; maybe not a lot? But seeing switchnull usually means something more serious is broken, which is pretty misleading in this case.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with package-manifest.toml and the dendrite dependency revision described in the issue; review dendrite PR #112 and the referenced omicron and Cargo.lock commits. Confirm that the updated dependency sends the api-version header to MGS, then verify that dpd switch-zone shell prompts show the connected switch instead of switchnull.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.