grpc / grpc/grpc-node

Alternate mechanism to `keepalive` for detecting broken connections

Open
#2,710 5 comments 0 reactions 0 assignees View on GitHub
package: @grpc/grpc-js
Dominant language
TypeScript
Stars
4.8k
Forks
716
Avg merge
2d 3h
Merged PRs (30d)
10

Description

### Is your feature request related to a problem? Please describe.
This [issue](https://github.com/grpc/grpc-node/issues/2664) describes why it is problematic to enable `keepalive` in serverless environments like AWS Lambda or Cloud Functions. Because of that, we turned off client-side keepalives only for serverless environments. That helped us eliminate all the sustained timeouts our clients were experiencing. However, sporadically (about twice a day), they still experience some `timeout` or `internal` errors. The logs added as a part of this [issue](https://github.com/grpc/grpc-node/issues/2687) were indeed helpful. In combination with those logs, we also perform a `getConnectivityState` in our interceptor when a request fails with `deadline exceeded`. Here's an example log from a client in production:

```
[2024-04-05T02:07:58.060Z] WARN (Momento: grpc-interceptor): Deadline Exceeded! Received status: 4 Deadline exceeded after 1.101s,remote_addr=34.192.43.23:443 and grpc connection status: SHUTDOWN
```

What I believe is happening is that the connection was broken, as indicated by `SHUTDOWN`. In the absence of `keepalive`, the client can only learn about the broken connection once it sends the request, so it is probably expected that the first request resulted in a `DEADLINE`. Given the caveat of enabling keepalive in serverless environments, is there an alternative for handling these? For clients whose requests are idempotent, this is not an issue as they can safely retry. However, some of our clients use `increment` APIs that we vend and aren't safe to retry.

### Describe the solution you'd like
- One possible solution is if we could if these operations are safe to retry. Other than that, I am looking for ideas as well :)

### Additional context
- https://github.com/grpc/grpc-node/issues/2664

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.