Integration tests are exhibiting flaky behavior

Seems to be failing more frequently:
```
=== RUN TestXdsClientGetsIncrementalResponsesFromUpstreamServer
2020/05/12 20:31:22 management server listening on 19001
TestXdsClientGetsIncrementalResponsesFromUpstreamServer: upstream_client_test.go:47:
Error Trace: upstream_client_test.go:47
Error: Setup failed: %s
Test: TestXdsClientGetsIncrementalResponsesFromUpstreamServer
Messages: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:19001: connect: connection refused"
```

And,
```
=== RUN TestServerShutdownShouldCloseResponseChannel
2020/05/12 18:43:45 listen tcp :19001: bind: address already in use
FAIL github.com/envoyproxy/xds-relay/integration 7.554s
FAIL
make: *** [integration-tests] Error 1
```