apollographql / apollographql/federation

Undici removeEventListener issue during outage

Open
#3,099 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
725
Forks
276
Avg merge
1h 47m
Merged PRs (30d)
1

Description

### Issue Description

Apollo outage on July 3rd caused issues communicating with uplink, forcing server crash.

We use undici as a primary fetch library for Apollo federation. We are in the process of integrating with Apollo enterprise, in our lower environment our server began to crash during a reported outage. The services recovered but our gateway crashed and impacted some testing in our lower environments. If we were using Apollo in production I would anticipate this would be a major outage for us.

Mainly asking to see if this is a known issue that has occurred for others, or if there's potentially an issue with using undici as a fetcher library.

Node version 18

```
/node_modules/undici/lib/web/fetch/request.js:57
this.removeEventListener('abort', abort)
^
TypeError: this.removeEventListener is not a function
at EventEmitter.abort (/node_modules/undici/lib/web/fetch/request.js:57:12)
at EventEmitter.emit (node:events:519:28)
at EventEmitter.emit (node:domain:488:12)
at AbortSignal.dispatchEvent (/node_modules/node-abort-controller/index.js:28:23)
at AbortController.abort (/node_modules/node-abort-controller/index.js:58:17)
at Timeout._onTimeout (/node_modules/@apollo/gateway/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts:130:16)
at listOnTimeout (node:internal/timers:573:17)
at processTimers (node:internal/timers:514:7)
```

Looking at the error source this appears to be related to a request timeout, which calls `node-abort-controller` polyfill which in turn tries to call the undici removeEventListener function which doesn't exist.

https://github.com/apollographql/federation/blob/38beaa2f753eaf31be74b272ea62f3c1948c854f/gateway-js/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts#L130

https://github.com/southpolesteve/node-abort-controller/blob/10e0cea66a069d9319f948d055621e1d37aea5db/index.js#L16

### Link to Reproduction

N/A

### Reproduction Steps

Service Dependencies:

```
"dependencies": {
"@apollo/gateway": "^2.8.3",
"@apollo/server": "^4.10.4",
"@aws-sdk/client-ssm": "^3.577.0",
"body-parser": "^1.20.2",
"cors": "^2.8.5",
"dd-trace": "^5.0.0",
"express": "^4.19.2",
"fs": "0.0.1-security",
"global-agent": "^3.0.0",
"http-status-codes": "^2.1.4",
"jsonwebtoken": "^9.0.0",
"lodash": "^4.17.21",
"nodejs-base64": "^2.0.0",
"superagent": "^8.0.0",
"swagger-ui-express": "^5.0.0",
"undici": "^6.16.1",
"winston": "^3.8.1",
"ws": ">=6.2.2",
"yamljs": "^0.3.0"
},
```

Gateway init:

```
import { fetch } from 'undici'
const gateway = new ApolloGateway({
fetcher: fetch,
logger: appLogger,
});
```

Node version: 18

Contributor guide

Open the contributing guide

Research direction

Start with gateway-js/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts around the timeout at line 130, then compare the abort handling in undici/lib/web/fetch/request.js with node-abort-controller/index.js. Reproduce the Node 18 timeout path if possible; done means the outage-related abort no longer crashes the gateway.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, node.js, typescript
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.