networking background tasks emit lots of warnings in simulated environments (including test suite)

It's easiest to see this in omicron-dev run-all:

$ cargo run --bin=omicron-dev -- run-all
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.50s
     Running `target/debug/omicron-dev run-all`
omicron-dev: setting up all services ... 
log file: /dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.0.log
note: configured to log to "/dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.0.log"
DB URL: postgresql://root@[::1]:43799/omicron?sslmode=disable
DB address: [::1]:43799
log file: /dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.2.log
note: configured to log to "/dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.2.log"
log file: /dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.3.log
note: configured to log to "/dangerzone/omicron_tmp/omicron-dev-omicron-dev.9781.3.log"
omicron-dev: services are running.
omicron-dev: nexus external API:    127.0.0.1:12220
omicron-dev: nexus internal API:    [::1]:12221
omicron-dev: cockroachdb pid:       9862
omicron-dev: cockroachdb URL:       postgresql://root@[::1]:43799/omicron?sslmode=disable
omicron-dev: cockroachdb directory: /dangerzone/omicron_tmp/.tmphbrhfC
omicron-dev: internal DNS HTTP:     http://[::1]:38018
omicron-dev: internal DNS:          [::1]:36531
omicron-dev: external DNS name:     oxide-dev.test
omicron-dev: external DNS HTTP:     http://[::1]:60088
omicron-dev: external DNS:          [::1]:50282
omicron-dev:   e.g. `dig @::1 -p 50282 test-suite-silo.sys.oxide-dev.test`
omicron-dev: management gateway:    http://[::1]:57986 (switch0)
omicron-dev: management gateway:    http://[::1]:58720 (switch1)
omicron-dev: silo name:             test-suite-silo
omicron-dev: privileged user name:  test-privileged

If you look at the log file, it's emitting lots of warnings. It's easiest to see them by filtering for warning-level messages:

23:06:56.891Z WARN e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c (ServerContext): failed to identify switch slot for dendrite, will retry in 2 seconds
    background_task = bfd_manager
    reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146)
    zone_address = ::1
23:06:58.007Z WARN e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c (ServerContext): failed to identify switch slot for dendrite, will retry in 2 seconds
    background_task = nat_v4_garbage_collector
    reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146)
    zone_address = ::1
23:06:59.285Z WARN e6bff1ff-24fb-49dc-a54e-c6a350cd4d6c (ServerContext): failed to identify switch slot for dendrite, will retry in 2 seconds
    background_task = switch_port_config_manager
    rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
    reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): error trying to connect: tcp connect error: Connection refused (os error 146)
    zone_address = ::1

This also happens if you run Nexus by hand and I expect it happens in the test suite, too.

These warnings are coming from at least three different background tasks.

This is coming from map_switch_zone_addrs():
https://github.com/oxidecomputer/omicron/blob/e4bcfeeef8b73d60fd880a4bce3cd2465cb11c65/nexus/src/app/mod.rs#L1036-L1041

I noticed that if you're running Nexus by hand, you run into the same warning and it blocks Nexus startup. The workaround seems to be to set mgd in the Nexus config file to point directly at the instances. That's what the test suite does:
https://github.com/oxidecomputer/omicron/blob/e4bcfeeef8b73d60fd880a4bce3cd2465cb11c65/nexus/test-utils/src/lib.rs#L548

and it works because it sets up clients directly:
https://github.com/oxidecomputer/omicron/blob/e4bcfeeef8b73d60fd880a4bce3cd2465cb11c65/nexus/src/app/mod.rs#L282-L288

and bypasses the loop that emits this warning:
https://github.com/oxidecomputer/omicron/blob/e4bcfeeef8b73d60fd880a4bce3cd2465cb11c65/nexus/src/app/mod.rs#L322

This might be a dup of #5201? I was confused that even after setting these values in the config, that fixed one part of Nexus (the startup path) but not the other (the background task). I guess maybe the difference is that the startup path was getting stuck on mgd, while the background tasks are getting stuck on dendrite, and I only overrode mgd in my config?

Start with map_switch_zone_addrs() in nexus/src/app/mod.rs and compare the configured mgd handling at the referenced lines 282-288 and 322. Review the test setup in nexus/test-utils/src/lib.rs and run omicron-dev run-all or Nexus manually to reproduce the warnings; done means simulated environments no longer emit repeated connection warnings or block startup.