E2E tests occasionally fail with `No connection could be made because the target machine actively refused it` errors

**Describe the bug**
E2E tests on the pipeline occasionally fail with the following error:

```
[2020-11-12T00:46:20.4899856Z] [WIN-CB3SCTO3KGE] [Exception] [JvmObjectId] Unable to read data from the transport connection: An established connection was aborted by the software in your host machine..
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.BufferedStream.Read(Byte[] array, Int32 offset, Int32 count)
at Microsoft.Spark.Interop.Ipc.SerDe.TryReadBytes(Stream s, Byte[] buffer, Int32 length) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\SerDe.cs:line 194
at Microsoft.Spark.Interop.Ipc.SerDe.ReadInt32(Stream s) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\SerDe.cs:line 75
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmBridge.cs:line 309
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object arg0) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmBridge.cs:line 138
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallStaticJavaMethod(String className, String methodName, Object arg0) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmBridge.cs:line 101
at Microsoft.Spark.Interop.Ipc.JvmObjectId.RemoveId() in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmObjectReference.cs:line 92
[2020-11-12T00:46:21.4795680Z] [WIN-CB3SCTO3KGE] [Exception] [JvmBridge] No connection could be made because the target machine actively refused it. 127.0.0.1:5567
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at Microsoft.Spark.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port, String secret) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Network\DefaultSocketWrapper.cs:line 85
at Microsoft.Spark.Interop.Ipc.JvmBridge.GetConnection() in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmBridge.cs:line 88
at Microsoft.Spark.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] args) in D:\a\1\s\dotnet-spark\src\csharp\Microsoft.Spark\Interop\Ipc\JvmBridge.cs:line 309

```

Need to look into why and how to work around these connection failures.
Example [pipeline here](https://dev.azure.com/dnceng/public/_build/results?buildId=883493&view=logs&j=d36646ff-4410-512d-eea3-2bc38581fe7c&t=5d205416-11bb-508b-2d07-46cad46bb24c)