memory leak in grpc rebalancer

#### Overview of the Issue

Noticed a suspected memory leak in our clusters

After looking into the heap dump (using `consul debug`) from two nodes with different boot time (several hours vs ~ a month),
here is one part we noticed big difference:

It looks like the underlying grpc lb is not releasing the stale connections.

#### Reproduction Steps

The _real_ way to reproduce is to let the node run for a long time, and notice the memory usage increase.

But here is a test case trying to reproduce the issue:

added to agent/grpc/client_test.go

```go
func TestClientConnPool_IntegrationWithGRPCResolver_Rebalance5000(t *testing.T) {
count := 5
res := resolver.NewServerResolverBuilder(newConfig(t))
registerWithGRPC(t, res)
pool := NewClientConnPool(ClientConnPoolConfig{
Servers: res,
UseTLSForDC: useTLSForDcAlwaysTrue,
DialingFromServer: true,
DialingFromDatacenter: "dc1",
})

for i := 0; i < count; i++ {
name := fmt.Sprintf("server-%d", i)
srv := newTestServer(t, name, "dc1", nil)
res.AddServer(srv.Metadata())
t.Cleanup(srv.shutdown)
}

conn, err := pool.ClientConn("dc1")
require.NoError(t, err)
client := testservice.NewSimpleClient(conn)

ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
t.Cleanup(cancel)

_, err = client.Something(ctx, &testservice.Req{})
require.NoError(t, err)

t.Run("rebalance the dc", func(t *testing.T) {
// Rebalance is random, but if we repeat it a few times it should give us a
// new server.
attempts := 5000
for i := 0; i < attempts; i++ {
res.NewRebalancer("dc1")()

_, err := client.Something(ctx, &testservice.Req{})
require.NoError(t, err)
}
t.Log("*** number of connections in mem:",
reflect.ValueOf(client).Elem().
FieldByName("cc").Elem().
FieldByName("conns").Len(), "***")
time.Sleep(10 * time.Second)
})
}
```

Output:

```
=== RUN TestClientConnPool_IntegrationWithGRPCResolver_Rebalance5000/rebalance_the_dc
client_test.go:345: *** number of connections in mem: 7946 ***
--- PASS: TestClientConnPool_IntegrationWithGRPCResolver_Rebalance5000 (11.87s)
```

`cd agent/grpc && go test -c && ./grpc.test -test.v -test.run '^TestClientConnPool_IntegrationWithGRPCResolver_Rebalance5000$'`

Change the `attempts` from 1 to 5000 increase memory of the testing program from 4mb to 28mb.
5000 is chosen to simulate the number of rebalances in 10 days ~ (60m * 24h * 10d / 2.5m), where 2.5m is the expected rebalance interval

### Consul info for both Client and Server

v1.10.3 linux amd64

Start with agent/grpc/client_test.go and the ClientConnPool rebalancer path, focusing on the repeated resolver rebalances and connection count. Run the provided grpc.test command to reproduce the growth. Done means repeated rebalancing no longer leaves stale connections accumulating or causes the observed memory increase.

Written by the indexing model from the issue text.