Unable to setup IPFS Swarm. failed to negotiate security protocol: context deadline exceeded.

#### Version information:
go-ipfs version: 0.4.22-dev-810cb607e-dirty
Repo version: 7
System version: amd64/linux
Golang version: go1.12.5

#### Description:
Unable to set up IPFS Swarm.

I am trying to deploy a n node swarm on bare metal machines on Grid5000.
I tried the same steps for n VM running on Vagrant, but I get the same error.

Did the following steps:
1. Build IPFS (See commit hash in version info)
2. `ipfs init`
3. generate swarm.key and copy it to `~/.ipfs/swarm.key` on all nodes. (I am using ipfs-swarm-key-gen)
4. `ipfs bootstrap rm --all`
5. `ipfs bootstrap add /ip4//tcp/4001/ipfs/` for all nodes. Here for n nodes, I am adding the multiaddr for the remaining n-1 nodes.

`ipfs swarm peers` returns empty.
`ipfs bootstrap` gives the n-1 nodes mutiadd.
`ipfs swarm addrs` gives the ` \n /ip4//tcp/4001` for the n-1 nodes.

I tried to telnet from one node to see if the port 4001 is accessible. It is accessible. Although, 8080 or 5001 doesn't work from outside the node.
I tried adding some files from one node, but `ipfs get ` never ends.

I found the following in the debug output of the daemon
`swarm2: got error on dial: failed to negotiate security protocol: context deadline exceeded swarm_dial.go:420`

`ipfs swarm connect ` gives the same error as above.

Can anyone help me with this.

Thank you