chaos of host member in a Original dst cluster

Background:
We are targeting to maintain thousands of hosts in a single original_dst cluster and we are seeing the flaws of the current original_dst cluster implementation.

From a high level, an original dst cluster maintains the active upstream hosts. The host members are created upon downstream requests.

Take a closer look,
1. The main thread maintains a host_map, keyed by host string address in the shape of "10.0.0.1".
2. The worker thread sees a snapshot of the host_map in the main thread, and each worker thread maintains a connpool keyed by Host object memory address. (0x7fdeadbeef)

The host_map in mainthread and the connpool are naturally out of sync because their keys are different.

The out-of-sync can occur when

* a worker thread can create a number of Host object before the host is posted to the mainthread and propagated to all worker threads.
* During the host GC, when the main thread decides to clean up the Host because of "not-active-requested"(note that there can be active upstream connections!), the host are erased from the host_map. The connections to this host won't be closed, not even drained.

So each worker thread may maintain an unlimited number of stale Hosts because there are connection pools within the stale hosts. Fortunately, the connpool are not leaked due to connpool is ref counted. See "envoy.reloadable_features.conn_pool_delete_when_idle".

Now that the host_map is stale anyway, I propose to maintain per worker thread independent active Host.

Pros:
1. Kill the main thread host GC. Currently the host is evicted if two consecutive scans mark the host inactive.
2. Avoid the copy-on-write host_map at main thread. It's not ideal to copy the rest of the host when the goal is to add or remove one of the hosts.
3. A better view of the current active hosts.
4. A predictable number of hosts: No more than ( # of concurrency) x (# of active hosts)

Cons:
1. A host with the same ip address can be duplicated in the worker threads.

Alternative

Protect the host_map access by a mutex. This mutex needs to be held when finding or adding a host at the worker thread during serving the request.