TestWebSocket* and TestGRPC* does not pass with parallel

## In what area(s)?

/area networking
/area test-and-release

As commented https://github.com/knative/serving/pull/11322/files#r636574573, e2e test with `--https` option does not work with `t.Parallel()` for TestWebSocket*.

Example log
https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/11322/pull-knative-serving-https/1395322718584311808
https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/11322/pull-knative-serving-https/1395530844940013568

It often fails as it cannot establish the connection as:

```
websocket.go:83: Connection failed: dial tcp 35.231.11.204:443: connect: connection refused
websocket_test.go:143: timed out waiting for the condition
```

FWIW, the error does not happen on local env (tested on AWS) by running `go test -v -race -tags=e2e -count=1 ./test/e2e/ -run "TestGRPC|TestWebSocket"` command.

Start by running `go test -v -race -tags=e2e -count=1 ./test/e2e/ -run "TestGRPC|TestWebSocket"` and inspect `test/e2e/`, especially `websocket.go` and `websocket_test.go`. Compare the HTTPS parallel failures with the linked Prow logs; done means TestWebSocket* and TestGRPC* pass reliably with `t.Parallel()` and `--https`.

Written by the indexing model from the issue text.