apollographql / apollographql/apollo-server

TCP RST and 502 errors when pods are terminated

Open
#7,935 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14k
Forks
2k
Avg merge
5h 30m
Merged PRs (30d)
1

Description

### Issue Description

We see 502 errors when an apollo-server pod is gracefully terminated in k8s. We host apollo-server in EKS and expose it using AWS ALBs.

AWS provides a [troubleshooting guide for 502s](https://repost.aws/knowledge-center/elb-alb-troubleshoot-502-errors) and we can see our issue falls under the criteria for "The load balancer received a TCP RST from the target when attempting to establish a connection" (see screenshot below).

We attempted to increase the `stopGracePeriodMillis` in `ApolloServerPluginDrainHttpServer` to be higher than the kubernetes `terminationGracePeriodSeconds` and the ALB Target Group `deregistration_delay` but did not see a change in behavior.

We also have set `httpServer.keepAliveTimeout` and `httpServer.headersTimeout` higher than our ALB Session Timeout.

![ALB Logs](https://github.com/user-attachments/assets/1831f52f-91b2-4aaa-8908-5262cb7fab41)

### Link to Reproduction

https://repost.aws/knowledge-center/elb-alb-troubleshoot-502-errors

### Reproduction Steps

1. Terminate a pod running running apollo-server in EKS behind an AWS ALB

Contributor guide

Open the contributing guide

Research direction

Start with ApolloServerPluginDrainHttpServer and reproduce pod termination in EKS behind an AWS ALB, using the linked AWS troubleshooting guide as context. Determine where the TCP RST and resulting 502 occur during shutdown; done means identifying a repository-level fix or documenting that the behavior depends on the deployment configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, node.js, typescript
Domain
backend, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.