github.com/hyperledger/fabric/gossip/service unit-test often fails on fabric v2.4.0

Environments
* OS: ubuntu 20.04.3
* golang: go version go1.16.8 linux/amd64
* hyperledger fabric v2.4.0

Procedures:
1. `git clone https://github.com/hyperledger/fabric.git`
2. Enter the fabric directory.
3. `git checkout v2.4.0`
4. `git revert 40f90c1feece5e9d696cedd960f2d132d37e18bb`
commit: 40f90c1fe Fix typo in comment (#3016)
(It changes only a comment in gossip/privdata/pvtdataprovider.go)
5. `export TEST_PKGS=github.com/hyperledger/fabric/gossip/service`
(for testing github.com/hyperledger/fabric/gossip/service only)
6. `make dist-clean clean all`
7. `make unit-test`
(20 times)

Expected: github.com/hyperledger/fabric/gossip/service always succeeds
Actual: Pass: 10, Fail: 11

```
--- FAIL: TestWithStaticDeliverClientLeader (46.02s)
gossip_service_test.go:628: Failed to establish full channel membership. Only 0 out of 2 peers have full membership
FAIL

--- FAIL: TestLeaderYield (30.03s)
gossip_service_test.go:628: Failed to establish full channel membership. Only 0 out of 2 peers have full membership
FAIL

--- FAIL: TestWithStaticDeliverClientNotLeader (46.02s)
gossip_service_test.go:628: Failed to establish full channel membership. Only 0 out of 2 peers have full membership
FAIL

--- FAIL: TestWithStaticDeliverClientBothStaticAndLeaderElection (46.02s)
gossip_service_test.go:628: Failed to establish full channel membership. Only 0 out of 2 peers have full membership
FAIL
```

Full Log
https://gist.github.com/justin-themedium/8562c733f49d7742b34e193544b34a5b