oxidecomputer / oxidecomputer/dendrite

tfport_ensure() failed to delete and recreate all VLANs and IPs

Open
#75 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
20
Forks
3
Avg merge
3d 11h
Merged PRs (30d)
4

Description

Originally reported as https://github.com/oxidecomputer/dendrite-archived/issues/650. No updates in nearly 2 years, so maybe it's been fixed as a side effect of something else?

On the dogfood rack during an update, folks reported missing IP interfaces and datalinks for some of the Gimlets. Specifically, Gimlet 9 was part of the cluster to be updated, but there appeared to be no gimlet9 IP interface. To start, here's the build we're running against, from swadm build-info.

root@oxz_switch:~# swadm build-info
Version: 0.2.0
Commit SHA: 363e365135cfa46d7f7558d8670f35aa8fe412e9
Commit timestamp: 2023-08-30T17:16:30.000000000Z
Git branch: main
SDE commit SHA: 46dee46717efb9b027555355d3293bfa4200293c
Rustc version: 1.72.0
Rustc channel: stable
Rustc triple: x86_64-unknown-illumos
Rustc commit SHA: 5680fa18feaa87f3ff04063800aec256c3d4b4be
Cargo triple: x86_64-unknown-illumos
Debug: false
Opt level: 3

Here are the IP addresses with gimlet in their name:

root@oxz_switch:~# ipadm | grep gimlet
gimlet1/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet1/10
gimlet3/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet3/10
gimlet4/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet4/10
gimlet5/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet5/10
gimlet6/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet6/10
gimlet7/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet7/10
gimlet8/ll        addrconf ok           fe80::aa40:25ff:fe05:2%gimlet8/10
gimlet10/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet10/10
gimlet11/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet11/10
gimlet12/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet12/10
gimlet14/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet14/10
gimlet15/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet15/10
gimlet16/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet16/10
gimlet17/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet17/10
gimlet18/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet18/10
gimlet19/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet19/10
gimlet20/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet20/10
gimlet21/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet21/10
gimlet22/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet22/10
gimlet23/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet23/10
gimlet25/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet25/10
gimlet26/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet26/10
gimlet27/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet27/10
gimlet28/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet28/10
gimlet29/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet29/10
gimlet30/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet30/10
gimlet31/ll       addrconf ok           fe80::aa40:25ff:fe05:2%gimlet31/10

It looks like we're missing gimlet{0,2,9,13,24} from this list. The datalinks appear to also be missing some entries, though not the exact same set.

root@oxz_switch:~# dladm | grep gimlet | sort
gimlet1     vlan      1500   up       --         tfportint0_0
gimlet10    vlan      1500   up       --         tfportint0_0
gimlet11    vlan      1500   up       --         tfportint0_0
gimlet12    vlan      1500   up       --         tfportint0_0
gimlet14    vlan      1500   up       --         tfportint0_0
gimlet15    vlan      1500   up       --         tfportint0_0
gimlet16    vlan      1500   up       --         tfportint0_0
gimlet17    vlan      1500   up       --         tfportint0_0
gimlet18    vlan      1500   up       --         tfportint0_0
gimlet19    vlan      1500   up       --         tfportint0_0
gimlet20    vlan      1500   up       --         tfportint0_0
gimlet21    vlan      1500   up       --         tfportint0_0
gimlet22    vlan      1500   up       --         tfportint0_0
gimlet23    vlan      1500   up       --         tfportint0_0
gimlet24    vlan      1500   up       --         tfportint0_0
gimlet25    vlan      1500   up       --         tfportint0_0
gimlet26    vlan      1500   up       --         tfportint0_0
gimlet27    vlan      1500   up       --         tfportint0_0
gimlet28    vlan      1500   up       --         tfportint0_0
gimlet29    vlan      1500   up       --         tfportint0_0
gimlet3     vlan      1500   up       --         tfportint0_0
gimlet30    vlan      1500   up       --         tfportint0_0
gimlet31    vlan      1500   up       --         tfportint0_0
gimlet4     vlan      1500   up       --         tfportint0_0
gimlet5     vlan      1500   up       --         tfportint0_0
gimlet6     vlan      1500   up       --         tfportint0_0
gimlet7     vlan      1500   up       --         tfportint0_0
gimlet8     vlan      1500   up       --         tfportint0_0

Here, we're missing gimlet{0,2,9,13}. Looking at the tfport log we can see some issues:

Dec 28 00:02:34.335 INFO created tfport for link, tfport: 2, link: tfportint0_0
Dec 28 00:02:34.374 INFO created vlan 265:gimlet12 on tfportint0_0
Dec 28 00:02:34.400 INFO created vlan 269:gimlet4 on tfportint0_0
Dec 28 00:02:34.424 INFO created vlan 281:gimlet13 on tfportint0_0
Dec 28 00:02:34.469 INFO created vlan 268:gimlet6 on tfportint0_0
Dec 28 00:02:34.494 INFO created vlan 287:gimlet3 on tfportint0_0
Dec 28 00:02:34.520 INFO created vlan 308:gimlet21 on tfportint0_0
Dec 28 00:02:34.546 INFO created vlan 285:gimlet5 on tfportint0_0
Dec 28 00:02:34.572 INFO created vlan 286:gimlet1 on tfportint0_0
Dec 28 00:02:34.599 INFO created vlan 272:gimlet25 on tfportint0_0
Dec 28 00:02:34.623 INFO created vlan 276:gimlet17 on tfportint0_0
Dec 28 00:02:34.647 INFO created vlan 257:gimlet26 on tfportint0_0
Dec 28 00:02:34.671 INFO created vlan 264:gimlet14 on tfportint0_0
Dec 28 00:02:34.714 INFO created vlan 275:gimlet29 on tfportint0_0
Dec 28 00:02:34.738 INFO created vlan 259:gimlet28 on tfportint0_0
Dec 28 00:02:34.763 INFO created vlan 271:gimlet2 on tfportint0_0
Dec 28 00:02:34.790 INFO created vlan 300:techport0 on tfportint0_0
Dec 28 00:02:34.816 INFO created vlan 301:techport1 on tfportint0_0
Dec 28 00:02:34.842 INFO created vlan 282:gimlet9 on tfportint0_0
Dec 28 00:02:34.866 INFO created vlan 284:gimlet7 on tfportint0_0
Dec 28 00:02:34.891 INFO created vlan 297:psc1 on tfportint0_0
Dec 28 00:02:34.916 INFO created vlan 305:tofino0 on tfportint0_0
Dec 28 00:02:34.946 INFO created vlan 266:gimlet8 on tfportint0_0
Dec 28 00:02:34.970 INFO created vlan 304:sidecar0 on tfportint0_0
Dec 28 00:02:34.995 INFO created vlan 280:gimlet15 on tfportint0_0
Dec 28 00:02:35.021 INFO created vlan 298:sidecar1 on tfportint0_0
Dec 28 00:02:35.047 INFO created vlan 283:gimlet11 on tfportint0_0
Dec 28 00:02:35.074 INFO created vlan 263:gimlet20 on tfportint0_0
Dec 28 00:02:35.101 INFO created vlan 296:psc0 on tfportint0_0
Dec 28 00:02:35.125 INFO created vlan 256:gimlet24 on tfportint0_0
Dec 28 00:02:35.151 INFO created vlan 261:gimlet18 on tfportint0_0
Dec 28 00:02:35.189 INFO created vlan 260:gimlet16 on tfportint0_0
Dec 28 00:02:35.214 INFO created vlan 258:gimlet30 on tfportint0_0
Dec 28 00:02:35.239 INFO created vlan 273:gimlet27 on tfportint0_0
Dec 28 00:02:35.264 INFO created vlan 270:gimlet0 on tfportint0_0
Dec 28 00:02:35.287 INFO created vlan 262:gimlet22 on tfportint0_0
Dec 28 00:02:35.313 INFO created vlan 274:gimlet31 on tfportint0_0
Dec 28 00:02:35.337 INFO created vlan 307:gimlet23 on tfportint0_0
Dec 28 00:02:35.362 INFO created vlan 277:gimlet19 on tfportint0_0
Dec 28 00:02:35.388 INFO created vlan 267:gimlet10 on tfportint0_0
Dec 28 00:02:35.388 DEBG Created packet queue for asic_id 2
Dec 28 00:02:35.409 DEBG creating interface for tfportint0_0
Dec 28 00:02:35.417 INFO dpd techport addressing setup complete
Dec 28 00:02:35.425 DEBG creating link-local address for tfportint0_0
Dec 28 00:02:35.752 INFO added new NDP entry, interface: tfportint0_0, entry: tfportint0_0:fe80::aa40:25ff:fe0d:1f24->a8:40:25:0d:1f:24
Dec 28 00:02:35.753 DEBG processing packet queue on 2 for a8:40:25:0d:1f:24
Dec 28 00:02:36.372 INFO illumos techport addressing setup complete
Dec 28 00:02:36.372 DEBG sending techport RA for fdb1:a840:2504:195:: from fe80::aa40:25ff:fe0d:1f24 on ifindex 19
Dec 28 00:02:36.373 DEBG sending techport RA for fdb2:a840:2504:195:: from fe80::aa40:25ff:fe0d:1f24 on ifindex 20
Dec 28 00:02:36.843 INFO sending new tfport address: fe80::aa40:25ff:fe0d:1f24
Dec 28 00:02:38.758 WARN tfport changed mac addresses, stale_mac: a8:40:25:0d:1f:24, mac: a8:40:25:05:00:02, tfport: tfportint0_0
Dec 28 00:02:38.793 INFO tfport found with stale data, stale_asic_id: 2, asic_id: 2, stale_mac: a8:40:25:0d:1f:24, mac: a8:40:25:05:00:02, tfport: tfportint0_0
Dec 28 00:02:38.794 DEBG cleaning up tfport tfportint0_0
Dec 28 00:02:38.846 INFO deleted vlan 256:gimlet24 on tfportint0_0
Dec 28 00:02:38.871 INFO deleted vlan 300:techport0 on tfportint0_0
Dec 28 00:02:38.910 INFO deleted vlan 285:gimlet5 on tfportint0_0
Dec 28 00:02:38.934 INFO deleted vlan 281:gimlet13 on tfportint0_0
Dec 28 00:02:38.958 INFO deleted vlan 266:gimlet8 on tfportint0_0
Dec 28 00:02:38.983 INFO deleted vlan 260:gimlet16 on tfportint0_0
Dec 28 00:02:39.006 INFO deleted vlan 287:gimlet3 on tfportint0_0
Dec 28 00:02:39.030 INFO deleted vlan 259:gimlet28 on tfportint0_0
Dec 28 00:02:39.055 INFO deleted vlan 258:gimlet30 on tfportint0_0
Dec 28 00:02:39.078 INFO deleted vlan 274:gimlet31 on tfportint0_0
Dec 28 00:02:39.103 INFO deleted vlan 307:gimlet23 on tfportint0_0
Dec 28 00:02:39.135 INFO deleted vlan 275:gimlet29 on tfportint0_0
Dec 28 00:02:39.159 INFO deleted vlan 267:gimlet10 on tfportint0_0
Dec 28 00:02:39.183 INFO deleted vlan 271:gimlet2 on tfportint0_0
Dec 28 00:02:39.207 INFO deleted vlan 261:gimlet18 on tfportint0_0
Dec 28 00:02:39.230 INFO deleted vlan 304:sidecar0 on tfportint0_0
Dec 28 00:02:39.255 INFO deleted vlan 280:gimlet15 on tfportint0_0
Dec 28 00:02:39.280 INFO deleted vlan 273:gimlet27 on tfportint0_0
Dec 28 00:02:39.303 INFO deleted vlan 301:techport1 on tfportint0_0
Dec 28 00:02:39.326 INFO deleted vlan 305:tofino0 on tfportint0_0
Dec 28 00:02:39.375 INFO deleted vlan 286:gimlet1 on tfportint0_0
Dec 28 00:02:39.399 INFO deleted vlan 284:gimlet7 on tfportint0_0
Dec 28 00:02:39.422 INFO deleted vlan 308:gimlet21 on tfportint0_0
Dec 28 00:02:39.446 INFO deleted vlan 277:gimlet19 on tfportint0_0
Dec 28 00:02:39.469 INFO deleted vlan 265:gimlet12 on tfportint0_0
Dec 28 00:02:39.493 INFO deleted vlan 268:gimlet6 on tfportint0_0
Dec 28 00:02:39.517 INFO deleted vlan 262:gimlet22 on tfportint0_0
Dec 28 00:02:39.540 INFO deleted vlan 270:gimlet0 on tfportint0_0
Dec 28 00:02:39.570 INFO deleted vlan 297:psc1 on tfportint0_0
Dec 28 00:02:39.593 INFO deleted vlan 283:gimlet11 on tfportint0_0
Dec 28 00:02:39.617 INFO deleted vlan 282:gimlet9 on tfportint0_0
Dec 28 00:02:39.641 INFO deleted vlan 263:gimlet20 on tfportint0_0
Dec 28 00:02:39.666 INFO deleted vlan 264:gimlet14 on tfportint0_0
Dec 28 00:02:39.691 INFO deleted vlan 272:gimlet25 on tfportint0_0
Dec 28 00:02:39.716 INFO deleted vlan 276:gimlet17 on tfportint0_0
Dec 28 00:02:39.740 INFO deleted vlan 296:psc0 on tfportint0_0
Dec 28 00:02:39.763 INFO deleted vlan 269:gimlet4 on tfportint0_0
Dec 28 00:02:39.792 INFO deleted vlan 257:gimlet26 on tfportint0_0
Dec 28 00:02:39.818 INFO deleted vlan 298:sidecar1 on tfportint0_0
Dec 28 00:02:39.818 DEBG Removed packet queue for asic_id 2
Dec 28 00:02:39.852 INFO removed stale NDP entry :fe80::aa40:25ff:fe0d:1f24->a8:40:25:0d:1f:24
Dec 28 00:02:39.855 INFO created tfport for link, tfport: 2, link: tfportint0_0
Dec 28 00:02:39.890 INFO created vlan 308:gimlet21 on tfportint0_0
Dec 28 00:02:39.914 INFO created vlan 268:gimlet6 on tfportint0_0
Dec 28 00:02:39.938 INFO created vlan 304:sidecar0 on tfportint0_0
Dec 28 00:02:39.962 INFO created vlan 265:gimlet12 on tfportint0_0
Dec 28 00:02:39.987 INFO created vlan 273:gimlet27 on tfportint0_0
Dec 28 00:02:40.022 INFO created vlan 262:gimlet22 on tfportint0_0
Dec 28 00:02:40.048 INFO created vlan 272:gimlet25 on tfportint0_0
Dec 28 00:02:40.075 INFO created vlan 275:gimlet29 on tfportint0_0
Dec 28 00:02:40.101 INFO created vlan 277:gimlet19 on tfportint0_0
Dec 28 00:02:40.127 INFO created vlan 286:gimlet1 on tfportint0_0
Dec 28 00:02:40.151 INFO created vlan 263:gimlet20 on tfportint0_0
Dec 28 00:02:40.176 INFO created vlan 259:gimlet28 on tfportint0_0
Dec 28 00:02:40.202 INFO created vlan 274:gimlet31 on tfportint0_0
Dec 28 00:02:40.232 INFO created vlan 284:gimlet7 on tfportint0_0
Dec 28 00:02:40.258 INFO created vlan 285:gimlet5 on tfportint0_0
Dec 28 00:02:40.283 INFO created vlan 305:tofino0 on tfportint0_0
Dec 28 00:02:40.309 INFO created vlan 264:gimlet14 on tfportint0_0
Dec 28 00:02:40.334 INFO created vlan 287:gimlet3 on tfportint0_0
Dec 28 00:02:40.362 INFO created vlan 280:gimlet15 on tfportint0_0
Dec 28 00:02:40.387 INFO created vlan 297:psc1 on tfportint0_0
Dec 28 00:02:40.413 INFO created vlan 300:techport0 on tfportint0_0
Dec 28 00:02:40.444 INFO created vlan 258:gimlet30 on tfportint0_0
Dec 28 00:02:40.469 INFO created vlan 269:gimlet4 on tfportint0_0
Dec 28 00:02:40.495 INFO created vlan 261:gimlet18 on tfportint0_0
Dec 28 00:02:40.521 INFO created vlan 267:gimlet10 on tfportint0_0
Dec 28 00:02:40.548 INFO created vlan 276:gimlet17 on tfportint0_0
Dec 28 00:02:40.574 INFO created vlan 257:gimlet26 on tfportint0_0
Dec 28 00:02:40.599 INFO created vlan 266:gimlet8 on tfportint0_0
Dec 28 00:02:40.624 INFO created vlan 301:techport1 on tfportint0_0
Dec 28 00:02:40.655 INFO created vlan 260:gimlet16 on tfportint0_0
Dec 28 00:02:40.682 INFO created vlan 283:gimlet11 on tfportint0_0
Dec 28 00:02:40.708 INFO created vlan 298:sidecar1 on tfportint0_0
Dec 28 00:02:40.734 INFO created vlan 296:psc0 on tfportint0_0
Dec 28 00:02:40.761 INFO created vlan 307:gimlet23 on tfportint0_0
Dec 28 00:02:40.797 ERRO tfport_ensure() failed: creating vlan 256:gimlet24

Caused by:
    create-addr failed: ipadm: Could not create address: Addrconf already in progress, tfport: tfportint0_0
Dec 28 00:02:40.812 INFO created tfport for link, tfport: 428, link: tfportrear0_0
Dec 28 00:02:40.812 DEBG Created packet queue for asic_id 428

So this appears to have created all the VLANs during initial startup, as part of the process of fetching the base MAC from the SP. It looks like all 32 VLANs for the Gimlet links are deleted. Then during their re-creation, tfport_ensure() failed because an IPv4 link-local address already existed on the interface.

So that error stack is part of this:

https://github.com/oxidecomputer/dendrite/blob/0933cb00fac6c4ede4e6f360ccc0204cf37ce9b8/tfportd/src/tfport.rs#L350-L372

We've created the tfport object itself, and we have no record of the interface index, so we attempt to call tfport_ensure_interface().

https://github.com/oxidecomputer/dendrite/blob/0933cb00fac6c4ede4e6f360ccc0204cf37ce9b8/tfportd/src/tfport.rs#L254-L276

That is checking if there exists an interface with the provided name, and creates one if not. Apparently addrconf is still in progress on this link, presumably because the kernel and / or NDP deamon still has some state associated with it and is trying to detect a conflicting local address.

At this point, since the show-if command indicated there is no interface, that addrconf process should stop at some point "soon". We need to keep retrying this step, inside either of these methods, until it succeeds.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names tfport_ensure() and provides ipadm, dladm, and tfport logs, but no source files or tests. Start by locating tfport_ensure() and tracing the cleanup and recreation sequence after the MAC-address change. Done means all expected VLANs, datalinks, and IP interfaces are present after that sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.