`sync_switch_configuration` complains about multiple address lot blocks in the `initial-infra` address block

Open
#10,182 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
38/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Quiet
Tech stack
rust
Domain
backend, databases

Research direction

Start with nexus/src/app/background/tasks/sync_switch_configuration.rs:1252 and lines 1270-1293, then inspect nexus/db-queries/src/db/datastore/address_lot.rs:311-316. Trace how initial-infra blocks populate RackNetworkConfig and whether sled-agent consumes infra_ip_first/infra_ip_last, using the IPv4/IPv6 example to determine expected behavior. Done means the block cardinality, ordering, and downstream handling are explicitly resolved and covered by relevant tests.

Written by the indexing model from the issue text.

Description

Poking around the Nexus logs on dogfood, I noticed this warning from the sync_switch_configuration background task:

02:48:10.775Z WARN 37090b68-aa4c-456d-bf2a-c8631fcff50c (ServerContext): more than one block assigned to infra lot
    background_task = switch_port_config_manager
    blocks = [
        AddressLotBlock { id: 2ada3a23-8cc0-4dad-a5d5-ba030b3a581a, address_lot_id: 59dac3c8-af15-4468-b4c6-f573db150e2e, first_address: V4(Ipv4Network { addr: 172.20.15.21, prefix: 32 }), last_address: V4(Ipv4Network { addr: 172.20.15.22, prefix: 32 }) },
        AddressLotBlock { id: e30373de-88b5-427c-acb7-65f896695e40, address_lot_id: 59dac3c8-af15-4468-b4c6-f573db150e2e, first_address: V6(Ipv6Network { addr: fd00:99::1, prefix: 128 }), last_address: V6(Ipv6Network { addr: fd00:99::ffff, prefix: 128 }) }
    ]
    file = nexus/src/app/background/tasks/sync_switch_configuration.rs:1252
    rack_id = de608e01-b8e4-4d93-b972-a7dbed36dd22

This is coming from this bit of code, where we ask the datastore for all the address lot blocks in the initial-infra address lot, but only expect to find one (and ignore any other than the first): https://github.com/oxidecomputer/omicron/blob/e9509985ef5767ab40a2b30c4e8908252d04810e/nexus/src/app/background/tasks/sync_switch_configuration.rs#L1270-L1293

I haven't dug into this, but some casual observations / questions:

  1. There's no order_by in the address_lot_blocks_by_name query, so we're (accidentally?) relying on CRDB to consistently return the same row first
  2. We only use this to populate the infra_ip_first and infra_ip_last fields in the RackNetworkConfig we push to the bootstore. Does sled-agent act on these fields? If so, what would it do if we happened to pick the IPv6 address lot block instead of the IPv4 one?
  3. Do we expect to have multiple address lot blocks in the initial-infra address lot? (It seems like either the configuration is wrong or the bg task is incorrect and needs to account for multiple blocks.)
Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/omicron

All issues in oxidecomputer/omicron

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.