Unexpected behavior when configured with listeners that fail to bind.

*Description*:

Envoy successfully starts when configured with listeners on ports that it fails to bind to, which results in unexpected behavior (see below).

*Repro steps*:

Port `:10001` fail - both connections are matched to the wildcard listener `0.0.0.0:10001`, even though more specific listener `127.0.0.1:10001` exists (or at least, was configured):
```
$ curl 192.168.9.1:10001
This server is listening on *:10001
```
```
$ curl 127.0.0.1:10001
This server is listening on *:10001
```

Port `:10002` fail - all connections to the wildcard listener `0.0.0.0:10002` are refused, because listener `127.0.0.1:10002` was configured first:

```
$ curl 192.168.9.1:10002
curl: (7) Failed to connect to 192.168.9.1 port 10002: Connection refused
```
```
$ curl 127.0.0.1:10002
This server is listening on 127.0.0.1:10002
```

*Admin and Stats Output*:

```
$ curl -s localhost:10000/listeners
["0.0.0.0:10001","127.0.0.1:10001","127.0.0.1:10002","0.0.0.0:10002"]
```
```
$ curl -s localhost:10000/stats | grep ^listener
listener.0.0.0.0_10001.downstream_cx_active: 0
listener.0.0.0.0_10001.downstream_cx_destroy: 2
listener.0.0.0.0_10001.downstream_cx_total: 2
listener.0.0.0.0_10001.http.whatever.downstream_rq_1xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_2xx: 2
listener.0.0.0.0_10001.http.whatever.downstream_rq_3xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_4xx: 0
listener.0.0.0.0_10001.http.whatever.downstream_rq_5xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_1xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_2xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_3xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_4xx: 0
listener.0.0.0.0_10002.http.whatever.downstream_rq_5xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_1xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_2xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_3xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_4xx: 0
listener.127.0.0.1_10001.http.whatever.downstream_rq_5xx: 0
listener.127.0.0.1_10002.downstream_cx_active: 0
listener.127.0.0.1_10002.downstream_cx_destroy: 1
listener.127.0.0.1_10002.downstream_cx_total: 1
listener.127.0.0.1_10002.http.whatever.downstream_rq_1xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_2xx: 1
listener.127.0.0.1_10002.http.whatever.downstream_rq_3xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_4xx: 0
listener.127.0.0.1_10002.http.whatever.downstream_rq_5xx: 0
listener.admin.downstream_cx_active: 1
listener.admin.downstream_cx_destroy: 1
listener.admin.downstream_cx_total: 2
listener.admin.http.admin.downstream_rq_1xx: 0
listener.admin.http.admin.downstream_rq_2xx: 1
listener.admin.http.admin.downstream_rq_3xx: 0
listener.admin.http.admin.downstream_rq_4xx: 0
listener.admin.http.admin.downstream_rq_5xx: 0
listener_manager.listener_added: 4
listener_manager.listener_create_failure: 2
listener_manager.listener_create_success: 16
listener_manager.listener_modified: 0
listener_manager.listener_removed: 0
listener_manager.total_listeners_active: 4
listener_manager.total_listeners_draining: 0
listener_manager.total_listeners_warming: 0
listener.0.0.0.0_10001.downstream_cx_length_ms: P0(nan,1) P25(nan,1.05) P50(nan,1.1) P75(nan,2.05) P90(nan,2.08) P95(nan,2.09) P99(nan,2.098) P99.9(nan,2.0998) P100(nan,2.1)
listener.127.0.0.1_10002.downstream_cx_length_ms: P0(nan,1) P25(nan,1.025) P50(nan,1.05) P75(nan,1.075) P90(nan,1.09) P95(nan,1.095) P99(nan,1.099) P99.9(nan,1.0999) P100(nan,1.1)
listener.admin.downstream_cx_length_ms: P0(nan,nan) P25(nan,nan) P50(nan,nan) P75(nan,nan) P90(nan,nan) P95(nan,nan) P99(nan,nan) P99.9(nan,nan) P100(nan,nan)
```

*Config*:

```
admin:
access_log_path: "/dev/null"
address:
socket_address: { address: 127.0.0.1, port_value: 10000 }

static_resources:
listeners:
# :10001 - wildcard, localhost
- name: wildcard_10001
address:
socket_address: { address: 0.0.0.0, port_value: 10001 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: whatever
codec_type: AUTO
route_config:
virtual_hosts:
- name: whatever
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body: { inline_string: "This server is listening on *:10001\n" }
http_filters:
- name: envoy.router
- name: localhost_10001
address:
socket_address: { address: 127.0.0.1, port_value: 10001 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: whatever
codec_type: AUTO
route_config:
virtual_hosts:
- name: whatever
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body: { inline_string: "This server is listening on 127.0.0.1:10001\n" }
http_filters:
- name: envoy.router
# :10002 - localhost, wildcard
- name: localhost_10002
address:
socket_address: { address: 127.0.0.1, port_value: 10002 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: whatever
codec_type: AUTO
route_config:
virtual_hosts:
- name: whatever
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body: { inline_string: "This server is listening on 127.0.0.1:10002\n" }
http_filters:
- name: envoy.router
- name: wildcard_10002
address:
socket_address: { address: 0.0.0.0, port_value: 10002 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
stat_prefix: whatever
codec_type: AUTO
route_config:
virtual_hosts:
- name: whatever
domains: ["*"]
routes:
- match: { prefix: "/" }
direct_response:
status: 200
body: { inline_string: "This server is listening on *:10002\n" }
http_filters:
- name: envoy.router
```

*Logs*:

```
[info][main] source/server/server.cc:402] starting main dispatch loop
[debug][main] source/server/worker_impl.cc:91] worker entering dispatch loop
[critical][config] source/server/listener_manager_impl.cc:613] listener 'localhost_10001' failed to listen on address '127.0.0.1:10001' on worker
[debug][config] source/server/listener_manager_impl.cc:655] begin remove listener: name=localhost_10001
[debug][config] source/server/listener_manager_impl.cc:663] unknown/locked listener 'localhost_10001'. no remove
[critical][config] source/server/listener_manager_impl.cc:613] listener 'wildcard_10002' failed to listen on address '0.0.0.0:10002' on worker
[debug][config] source/server/listener_manager_impl.cc:655] begin remove listener: name=wildcard_10002
[debug][config] source/server/listener_manager_impl.cc:663] unknown/locked listener 'wildcard_10002'. no remove
```