Alternate mechanism to `keepalive` for detecting broken connections
- Dominant language
- TypeScript
- Stars
- 4.8k
- Forks
- 716
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
### Is your feature request related to a problem? Please describe.
This [issue](https://github.com/grpc/grpc-node/issues/2664) describes why it is problematic to enable `keepalive` in serverless environments like AWS Lambda or Cloud Functions. Because of that, we turned off client-side keepalives only for serverless environments. That helped us eliminate all the sustained timeouts our clients were experiencing. However, sporadically (about twice a day), they still experience some `timeout` or `internal` errors. The logs added as a part of this [issue](https://github.com/grpc/grpc-node/issues/2687) were indeed helpful. In combination with those logs, we also perform a `getConnectivityState` in our interceptor when a request fails with `deadline exceeded`. Here's an example log from a client in production:
```
[2024-04-05T02:07:58.060Z] WARN (Momento: grpc-interceptor): Deadline Exceeded! Received status: 4 Deadline exceeded after 1.101s,remote_addr=34.192.43.23:443 and grpc connection status: SHUTDOWN
```
What I believe is happening is that the connection was broken, as indicated by `SHUTDOWN`. In the absence of `keepalive`, the client can only learn about the broken connection once it sends the request, so it is probably expected that the first request resulted in a `DEADLINE`. Given the caveat of enabling keepalive in serverless environments, is there an alternative for handling these? For clients whose requests are idempotent, this is not an issue as they can safely retry. However, some of our clients use `increment` APIs that we vend and aren't safe to retry.
### Describe the solution you'd like
- One possible solution is if we could if these operations are safe to retry. Other than that, I am looking for ideas as well :)
### Additional context
- https://github.com/grpc/grpc-node/issues/2664
Contributor guide
Assessment
This issue has not been assessed yet.