Communication Analyzer

When an instance running on Oxide is communicating with another entity, there are a number of hops that communication goes through, each of which does some sort of processing on the individual packets that comprise the communication. This is true whether the instance is communicating with another instance within the rack, or some host outside the rack. Along the way a number of actions are taken on packets such as source address selection, firewall rule application, nexthop or boundary tunnel selection, etc. For users and operators that are not intimately familiar with the design and implementation of the Oxide platform this can feel opaque and when communications are not working, there is no clear direction of where to look.

Providing a communication analyzer could really help here. Allowing users to ask questions like

Tell me everything you know about a packet going from myproject/myinstance to https://example.com/api/foo.

This would result in something like this.

example.com resolves to 203.0.113.5

When packets first leave myproject/myinstance a source address will be selected from the instance.
The addresses assigned to the instance are the following

• 198.51.100.23 (floating IP)
• 198.51.100.12 (ephemeral)
• 198.51.100.87 (SNAT)

These addresses all come from the same IP pool default. There is a single default route in the
myproject VPC that points to the internet gateway default. The default internet gateway
is linked to the default IP pool. That means any of these source addresses may be used. When
multiple source IP addresses are available, floating IPs are used first. In this case 198.51.100.23
will be the source address used for the communication.

Now that the source address has been selected, the packet will undergo network address translation
NAT which will set 198.51.100.23 as the source address of the packet leaving the VM in place of the
VMs internal VPC address 172.30.0.3.

The next thing that will happen to this packet is firewall rule application. The following VPC firewall
rules apply to this packet

<firewall rules listed here>

None of these firewall rules will prevent packets from reaching example.com on port 443. When
response traffic returns from example.com to myproject/myinstance, even though there are no
specific ingress rules allowing the traffic to come through, because this traffic is a part of a session
initiated from within the rack to an external host, the responses will be allowed through for this
established connection. It looks like there is currently an established session to example.com from
myproject/myinstance, with the following statistics

<stats go here>

Next this packet will be encapsulated onto an overlay network for this project's VPC and sent to one
of the rack switches for egress to the upstream network.

Both of the switches on this rack have routes that provide a path to example.com. When the first
packet is sent as a part of a TCP session to example.com the Oxide Packet Transformation Engine
(OPTE) will select one of the switches as a gateway router. All subsequent packets that are a part of
that TCP session will use that switch as an exit point from the rack, assuming that switch keeps a route
to 198.51.100.23. In the event that a route to 198.51.100.23 is lost, or the selected switch becomes
unavailable, the path will shift to the other switch. <Insert warning here for non-HA configurations involving only one switch as an exit point>. It looks like there is currently a TCP session going from
myproject/myinstance to example.com. That traffic is currently pinned to switch1.

This is just a starting point. There is actually a lot more information that could be provided that would give users potentially useful insights into how the Oxide platform works and how that relates to the applications they've deployed.

This is also already quite wordy, and perhaps a flow diagram with annotations would be better.

The issue names no files, tests, or entry points. Start by defining a scoped analyzer design and its user-facing entry point from the proposed packet-flow questions; done should include agreed requirements and acceptance criteria for the information and paths it must report.