Automated e2e network configuration testing

Omicron #4585 introduces very basic automated testing for the a4x2 topology in CI. However, this needs to expand beyond simple "does it work" tests.

Update: automated testing need to be removed due to overloading CI resources. We'll need to solve that problem to make forward progress here

Update 2: More bare metal resources have been added to CI so we can start to look at enabling the a4x2 jobs again.

Update 3: Originally this issue was focused on a4x2 as a testing substrate, but we really need to be thinking about this for the simulated omicron environment, a4x2 and racklettes. The broadest possible set of tests should run on each platform (obviously some platforms can not run certain tests, like mupdate tests) to allow folks to efficiently run tests in the environments they are working in.

The following is a list of things we should cover the following scenarios to start with. To the extent possible each scenario should be captured as an API driven workflow and a terraform driven workflow. We can use this effort as a gateway to https://github.com/oxidecomputer/docs/issues/577.

Networking

• Probe all sleds on rack startup.
• Adding and removing switch port/link configuration.
• Adding, removing and updating BFD configuration.
• Adding, removing and updating BGP configuration.
• Adding, removing and updating static routes.
• Ensuring mupdate works across versions
• Scrimlet cold boot

Tests from other categories to be added here.