[Question] grpc + AWS NLB problems

### **[Problem]**
We're having a problem where load balancer ( nlb ) is not able to balance the requests to servers. I tried a few different approaches and it doesn't work as expected. Can someone advise?

### [Architecture Overview]
- We use ECS to run our containerized applications where we have around 11 services communicating with each other using unary rpc calls. We can run one or more instances of the same service and nlb should balance the load for us.

- We create and memoize the client/stub only once. ( we were creating one client per request and closing it after request resolved, but as far as I can see it's not the suggested approach ).

### **[Version]**
1.18.0

### **[Scenario 1]**

We set the server to expire the connection every 10 seconds and keep the connection alive at least 5 seconds.

PS.: The same happens if I remove all `keepalive` options and leave only `max_age`

Server options:
```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600,
'grpc.max_connection_age_ms': 10000,
'grpc.keepalive_time_ms': 5000,
'grpc.keepalive_timeout_ms': 1000,
'grpc.keepalive_permit_without_calls': 1
}
```
Client options:
```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600,
'grpc.keepalive_time_ms': 5000,
'grpc.keepalive_timeout_ms': 1000,
'grpc.keepalive_permit_without_calls': 1
}
```

With `GRPC_TRACE=all GRPC_VERBOSITY=DEBUG` I see a few GOAWAY logs that actually does not cause problems:
```
49 connectivity_state.cc:164] SET: 0x5633fa6e25e8 client_transport: READY --> TRANSIENT_FAILURE [got_goaway] error=0x5633fa29f4c0
{
"created": "@1562782391.092019273",
"description": "GOAWAY received",
"file": "../deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc",
"file_line": 1126,
"http2_error": 0,
"raw_bytes": "max_age"
}
```

Eventually ( once in a while ), I see a sequence of errors that finally throws a INTERNAL GOAWAY:
```
{
"created": "@1562782391.100345834",
"description": "Delayed close due to in-progress write",
"file": "../deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc",
"file_line": 586,
"referenced_errors": [
{
"created": "@1562782391.100333243",
"description": "Endpoint read failed",
"file": "../deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc",
"file_line": 2491,
"grpc_status": 14,
"occurred_during_write": 0,
"referenced_errors": [
{
"created": "@1562782391.100148045",
"description": "EOF",
"file": "../deps/grpc/src/core/lib/iomgr/tcp_uv.cc",
"file_line": 107
},
{
"created": "@1562782391.092019273",
"description": "GOAWAY received",
"file": "../deps/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc",
"file_line": 1126,
"http2_error": 0,
"raw_bytes": "max_age"
}
]
}
]
}
```

### **[Scenario 2]**

We removed `max-age` and decided we would keep `keepalive` with the hope that somehow grpc would take care of balancing the requests among healthy servers.

We don't get errors here but clients stick with the same server forever of course.

Server options:
```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600,
'grpc.keepalive_time_ms': 5000,
'grpc.keepalive_timeout_ms': 1000,
'grpc.keepalive_permit_without_calls': 1
}
```
Client options:
```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600,
'grpc.keepalive_time_ms': 5000,
'grpc.keepalive_timeout_ms': 1000,
'grpc.keepalive_permit_without_calls': 1
}
```

### **[Scenario 3]**

No `keepalive` or `max-age` here. However, the load balance will only have the chance to balance the request if nlb reaches its idle timeout. Meaning we're not balancing requests if they are coming in a large volume at once?

```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600
}
```
Client options:
```javascript
{
'grpc.max_send_message_length': 104857600,
'grpc.max_receive_message_length': 104857600
}
```

We're now getting sporadic `message=14 UNAVAILABLE: TCP Read failed` errors that were not happening with grpc-node@1.4.1
```
error: message=14 UNAVAILABLE: TCP Read failed, name=Error, code=14, , details=TCP Read failed, isOperational=true, stack=Error: 14 UNAVAILABLE: TCP Read failed
```