Creating a link via `apply_port_settings` behaves differently than `link_create -> link_enabled_set`

Originally reported by @jgallagher as https://github.com/oxidecomputer/dendrite-archived/issues/588

On madrid, if I attempt to create a link via swadm, which internally calls link_create and link_enabled_set for each of these two commands:

root@oxz_switch:~# swadm link create qsfp0 --fec none --speed 40G
Created link qsfp0/0
root@oxz_switch:~# swadm link set-prop qsfp0/0 ena true

the link does not come up. If I pass --autoneg when creating the link:

root@oxz_switch:~# swadm link create qsfp0 --fec none --speed 40G --autoneg
Created link qsfp0/0
root@oxz_switch:~# swadm link set-prop qsfp0/0 ena true

the link does come up.

However, if I use the apply_port_settings API endpoint (which is not exposed via swadm, so we use curl):

root@oxz_switch:~# cat /tmp/manual.json
{
    "tag": "manual",
    "links": {
        "0": {
            "params": {
                "speed": "Speed40G",
                "fec": "None",
                "autoneg": false,
                "kr": false
            },
            "addrs": []
        }
    },
    "v4_routes": {},
    "v6_routes": {}
}
root@oxz_switch:~# curl -H "Content-Type: application/json" -X POST http://[::1]:12224/port/qsfp0/settings -d @/tmp/manual.json

the link comes up regardless of whether "autoneg" in the JSON body is true or false.

Pasting some comments from @bnaecker when we were looking at this:

Reading the port-settings-apply stuff, it's certainly taking a different code path than when you do swadm link create. When you add a new link through the settings code path, that seems to go to dpd/src/port_settings.rs:571, which calls SwitchPorts::insert_link() with the new link. That's really an internal method to add the data to the set of ports.

When you call swadm link create, that starts at dpd::Switch::create_link(). That does a whole bunch of stuff first, mostly telling the SDE about the new link.

It looks like the settings apply code does a similar thing through the add_asic_layer_port() method at dpd/src/port_settings.rs:853. I see that doing most things, but notably seems to be leaving out calls to port_kr_set(), port_autoneg_set(), and I think also mac_set(). I'm not sure what happens if you never tell the SDE what those parameters should be.

Our hypothesis is that (perhaps?) the SDE is defaulting autoneg to true, and the fact that we omit a call to port_autoneg_set() in the apply_port_settings code path is a happy accident that causes this particular link (which requires autoneg to come up) to work.

A cautionary note: if we "fix" this before fixing https://github.com/oxidecomputer/omicron/issues/3061, it's possible we'll break any uplinks that actually do require autonegotiation (since the control plane is currently always passing autoneg=false).

Compare dpd/src/port_settings.rs at the settings-apply paths around insert_link() and add_asic_layer_port() with dpd::Switch::create_link(). Trace whether port_kr_set(), port_autoneg_set(), and mac_set() are intentionally omitted, then verify the expected behavior against the autonegotiation concern in omicron issue 3061. Done means the two creation paths have consistent, confirmed parameter behavior without breaking required uplinks.