Keepalive PING after graceful GOAWAY disconnects draining RPCs
- Dominant language
- TypeScript
- Stars
- 4.8k
- Forks
- 716
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 10
Description
### What version of gRPC and what language are you using?
`@grpc/grpc-js` 1.14.4 on Node.js 24.15.0. The relevant path is also unchanged on the current `master` branch.
### What operating system are you using?
Linux.
### What did you do?
Send a graceful HTTP/2 GOAWAY while a unary RPC is still running, then allow the RPC to finish after the client's keepalive interval.
This is closely related to #2624, but the current failure does not depend on an immediate server shutdown. A graceful GOAWAY by itself leaves the keepalive timer running. The next PING fails, `maybeSendPing()` calls `handleDisconnect()`, and grpc-js terminates the accepted in-flight RPC.
`goaway-server.js`:
```js
const http2 = require('node:http2');
function frameMessage(payload) {
const frame = Buffer.allocUnsafe(5 + payload.length);
frame.writeUInt8(0, 0);
frame.writeUInt32BE(payload.length, 1);
payload.copy(frame, 5);
return frame;
}
const server = http2.createServer();
server.on('stream', stream => {
stream.respond(
{ ':status': 200, 'content-type': 'application/grpc' },
{ waitForTrailers: true }
);
stream.on('wantTrailers', () => {
stream.sendTrailers({ 'grpc-status': '0' });
});
setTimeout(() => {
stream.session.goaway(http2.constants.NGHTTP2_NO_ERROR, stream.id);
}, 10);
setTimeout(() => {
stream.end(frameMessage(Buffer.from('response')));
}, 20_000);
});
server.listen(50051, '127.0.0.1');
```
`client.js`:
```js
const grpc = require('@grpc/grpc-js');
const client = new grpc.Client(
'127.0.0.1:50051',
grpc.credentials.createInsecure(),
{
'grpc.keepalive_time_ms': 15_000,
'grpc.keepalive_timeout_ms': 30_000,
'grpc.keepalive_permit_without_calls': 1
}
);
const startedAt = Date.now();
client.makeUnaryRequest(
'/repro.Service/Slow',
value => value,
value => value,
Buffer.from('request'),
{ deadline: Date.now() + 120_000 },
(error, response) => {
console.log({
elapsedMs: Date.now() - startedAt,
code: error?.code,
details: error?.details,
response: response?.toString()
});
client.close();
}
);
```
Run `node goaway-server.js`, then `node client.js`.
The same reproduction can be sped up deterministically by using a 100 ms keepalive and a 500 ms response delay.
### What did you expect to see?
The accepted RPC finishes normally after the graceful GOAWAY and returns `response` at about 20 seconds. The transport is already draining and will not accept a new call, so it should not send more keepalive PINGs.
### What did you see instead?
Stock 1.14.4 fails at the keepalive boundary (15,022 ms in the original reproduction) with status 14 / `Connection dropped`. In the scaled reproduction, stock 1.14.4 fails at about 114 ms; a transport that stops keepalive on GOAWAY returns the response at about 524 ms.
There is a second race when a PING is already in flight as GOAWAY arrives: a later PING cancellation callback can still call `handleDisconnect()` unless the callback also recognizes the draining state.
### Suggested fix
When the session emits `goaway`:
1. Mark the transport as draining.
2. Clear the keepalive timeout.
3. Make `canSendPing()` / `maybeSendPing()` reject future PINGs.
4. Ignore completion of a PING that was already in flight once the transport is draining.
The existing socket `close` and `error` handlers still report genuine disconnects, and RPC deadlines continue to bound hung calls. This only stops probing a connection whose peer explicitly announced graceful draining.
Contributor guide
Research direction
Start at the transport's goaway handling and trace canSendPing(), maybeSendPing(), the keepalive timeout, and the in-flight PING completion callback. Run the scaled reproduction with a 100 ms keepalive and 500 ms response delay; done means graceful GOAWAY stops future keepalive failure handling while the accepted RPC returns its response, without hiding socket close or error disconnects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend-api-design, networking
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100