`AddressAlreadyInUse` when using `WithDefaultEndPoint` but server starts anyway.

### Describe the bug
Upgrading from 3.x to the latest 4.x, I wanted to create a server without encryption. (for local use only)

Original 3.x code:

```csharp
var options = new MqttServerOptionsBuilder().WithConnectionBacklog(10)
.WithDefaultEndpointPort(32700)
.WithDefaultEndpointBoundIPAddress(System.Net.IPAddress.Loopback)
.WithApplicationMessageInterceptor(HandleApplicationMessage)
.WithSubscriptionInterceptor(HandleSubscription)
.Build();
```

New 4.x code:
```csharp
var options = new MqttServerOptionsBuilder().WithConnectionBacklog(10)
.WithoutEncryptedEndpoint()
.WithDefaultEndpoint() //To enable endpoint without encryption
.WithDefaultEndpointPort(32700)
.WithDefaultEndpointBoundIPAddress(System.Net.IPAddress.Loopback)
.Build();
```

As soon as `await server.StartAsync();` is called the 4.x version throws the following exception:

> Failed to start the broker: System.Net.Sockets.SocketException (0x80004005): Address already in use

However, the server is still started and clients can connect to it.
This is the only instance of the server and this code is only called once.
If `await server.StartAsync();` is commented out, the server is not started (obviously).
There are no other processes on the system using that port.

### Which component is your bug related to?

- Server

### To Reproduce
Steps to reproduce the behavior:
1. Using this version of MQTTnet '4.1.3.436'.
2. Run this code: The 4.x code above
3. With these arguments 'WithDefaultEndpoint'
4. See error: SocketException (but server starts anyway)

### Expected behavior
Expected it to start the server without an exception since the port is not in use

### Additional context / logging
- This is running on an isolated embedded system.
- It works perfectly with 3.x
- After catching the exception, clients can connect normally
- Clients run on the embedded system. There is no outside communication
- Nothing else is using the port
- Changing the port to a different value does not resolve the issue
- OS = Android (Custom BSP, completely under our control, so no restrictions)
- Framework = Mono

Start with the supplied MQTTnet 4.1.3.436 reproduction and trace await server.StartAsync() after WithDefaultEndpoint and the endpoint configuration calls. Compare startup behavior on the reported Mono/Android setup with the expected single bind on port 32700. Done means startup completes without AddressAlreadyInUse while clients can still connect.

Written by the indexing model from the issue text.