oxidecomputer / oxidecomputer/maghemite

ddm: Tunnel route reconciler should account for V2B entries present on initialisation

Open
#486 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug ddm
Dominant language
Rust
Stars
94
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
12

Description

Earlier today @askfongjojo hit some odd behaviour on berlin where ~50% of API operations would timeout. From there she verified that both switch zones on the racklette could communicate with the upstream switch. Altering the --source-port on curl allowed us to reproduce requests which would always succeed/fail, suggesting to us there was a path-sensitive issue.

In OPTE, we saw that nexus was indeed responding to the initial TCP SYN:

BRM42220011 # opteadm dump-tcp-flows -p opte0
FLOW                                   STATE     HITS  SEGS IN  SEGS OUT  BYTES IN  BYTES OUT
TCP:172.30.2.6:443:172.20.17.94:30001  SYN_RCVD  7     1        7         152       518

From there we looked at the V2B mappings -- OPTE was indeed selecting one, otherwise the outbound packets would have been dropped. We'd expected to see two such entries:

root@oxz_switch1:~# pilot host exec -c '/opt/oxide/opte/bin/opteadm dump-v2b' '1-31'                                                                                                [6/1884]
14  BRM42220023        ok: Virtual to Boundary Mappings
======================================================================

IPv4 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP             VNI
0.0.0.0/0        fd0a:5f09:b56:161d::1   99
0.0.0.0/0        fd15:1d04:84de:99f1::1  99
0.0.0.0/0        fdb3:b877:ed64:7b18::1  99
0.0.0.0/0        fdf0:896b:23f6:cd7::1   99

IPv6 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP  VNI
15  BRM42220011        ok: Virtual to Boundary Mappings
======================================================================

IPv4 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP             VNI
0.0.0.0/0        fd0a:5f09:b56:161d::1   99
0.0.0.0/0        fd15:1d04:84de:99f1::1  99
0.0.0.0/0        fdb3:b877:ed64:7b18::1  99
0.0.0.0/0        fdf0:896b:23f6:cd7::1   99

IPv6 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP  VNI
16  BRM42220082        ok: Virtual to Boundary Mappings
======================================================================

IPv4 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP             VNI
0.0.0.0/0        fd0a:5f09:b56:161d::1   99
0.0.0.0/0        fd15:1d04:84de:99f1::1  99

IPv6 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP  VNI
17  BRM06240029        ok: Virtual to Boundary Mappings
======================================================================

IPv4 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP             VNI
0.0.0.0/0        fd0a:5f09:b56:161d::1   99
0.0.0.0/0        fd15:1d04:84de:99f1::1  99
0.0.0.0/0        fdb3:b877:ed64:7b18::1  99
0.0.0.0/0        fdf0:896b:23f6:cd7::1   99

IPv6 mappings
----------------------------------------------------------------------
TUNNELED PREFIX  BOUNDARY IP  VNI

...except sleds 14, 15, and 17 have extra V2B mappings. Do these correspond to anything in the switch state?

root@oxz_switch1:~# swadm addr list
Link      IPv4          IPv6
int0/0                  fdb1:a840:2504:190::1
                        fdb2:a840:2504:190::1
                        fe80::aa40:25ff:fe05:1f02
rear0/0                 fe80::aa40:25ff:fe05:1f03
...
qsfp0/0   172.20.15.70
loopback  fd15:1d04:84de:99f1::1

root@oxz_switch0:~# swadm addr list
Link      IPv4          IPv6
int0/0                  fdb1:a840:2504:110::1
                        fdb2:a840:2504:110::1
                        fe80::aa40:25ff:fe05:3502
rear0/0                 fe80::aa40:25ff:fe05:3503
...
qsfp0/0   172.20.15.69
loopback  fd0a:5f09:b56:161d::1

They don't. The loopback addresses here are present in all sets of V2B mappings, and the only ones installed on sled 16. Nexus, in this case, was serving our requests from sled 15. If we look at the responses to the TCP SYNs, they're bound for one of those garbage addresses:

ETHER:  ----- Ether Header -----
ETHER:
ETHER:  Packet 10 arrived at 19:21:56.34162
ETHER:  Packet size = 144 bytes
ETHER:  Destination = a8:40:25:5:1f:12,
ETHER:  Source      = a8:40:25:4:2:19,
ETHER:  Ethertype = 86DD (IPv6)
ETHER:
IPv6:   ----- IPv6 Header -----
IPv6:
IPv6:   Version = 6
IPv6:   Traffic Class = 0
IPv6:   Flow label = 0x0
IPv6:   Payload length = 90
IPv6:   Next Header = 17 (UDP)
IPv6:   Hop Limit = 128
IPv6:   Source address = fd00:1122:3344:103::1
IPv6:   Destination address = fdf0:896b:23f6:cd7::1
IPv6:
UDP:  ----- UDP Header -----
UDP:
UDP:  Source port = 26019
UDP:  Destination port = 6081
UDP:  Length = 90
UDP:  Checksum = 0000 (no checksum)
UDP:

What does DDM think?

root@oxz_switch1:~# pilot host exec -c '/opt/oxide/mg-ddm/ddmadm tunnel-imported' '1-31'
14  BRM42220023        ok: Overlay Prefix  Boundary Address        VNI  Metric
0.0.0.0/0       fd15:1d04:84de:99f1::1  99   18446744073709551615
0.0.0.0/0       fd0a:5f09:b56:161d::1   99   18446744073709551615
15  BRM42220011        ok: Overlay Prefix  Boundary Address        VNI  Metric
0.0.0.0/0       fd15:1d04:84de:99f1::1  99   18446744073709551615
0.0.0.0/0       fd0a:5f09:b56:161d::1   99   18446744073709551615
16  BRM42220082        ok: Overlay Prefix  Boundary Address        VNI  Metric
0.0.0.0/0       fd15:1d04:84de:99f1::1  99   18446744073709551615
0.0.0.0/0       fd0a:5f09:b56:161d::1   99   18446744073709551615
17  BRM06240029        ok: Overlay Prefix  Boundary Address        VNI  Metric
0.0.0.0/0       fd15:1d04:84de:99f1::1  99   18446744073709551615
0.0.0.0/0       fd0a:5f09:b56:161d::1   99   18446744073709551615

root@oxz_switch1:~# ddmadm tunnel-originated
Overlay Prefix  Boundary Address        VNI  Metric
0.0.0.0/0       fd15:1d04:84de:99f1::1  99   18446744073709551615
root@oxz_switch0:~# ddmadm tunnel-originated
Overlay Prefix  Boundary Address       VNI  Metric
0.0.0.0/0       fd0a:5f09:b56:161d::1  99   18446744073709551615

It has the right idea, but it has not ensured those are the only TEPs installed. My current theory is that those were the switch addresses of an earlier incarnation of the control plane before the most recent clean-slate/RSS. We should be verifying against the state which is actually resident in OPTE, using the DumpV2b ioctl. While this is maybe an artefact of repeated clean-slates/RSS applied to the same rack, in practice we could end up in a similar spot if ddm crashes and restarts at the same time as other TEPs are installed or removed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DDM tunnel route reconciler and the DumpV2b ioctl mentioned in the report; compare OPTE-resident mappings with tunnel-imported and tunnel-originated. Done means reconciliation removes V2B entries that are not represented by current switch state, including after a DDM restart or concurrent TEP changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.