apollographql / apollographql/federation
Undici removeEventListener issue during outage
- Dominant language
- TypeScript
- Stars
- 725
- Forks
- 276
- Avg merge
- 1h 47m
- Merged PRs (30d)
- 1
Description
### Issue Description
Apollo outage on July 3rd caused issues communicating with uplink, forcing server crash.
We use undici as a primary fetch library for Apollo federation. We are in the process of integrating with Apollo enterprise, in our lower environment our server began to crash during a reported outage. The services recovered but our gateway crashed and impacted some testing in our lower environments. If we were using Apollo in production I would anticipate this would be a major outage for us.
Mainly asking to see if this is a known issue that has occurred for others, or if there's potentially an issue with using undici as a fetcher library.
Node version 18
```
/node_modules/undici/lib/web/fetch/request.js:57
this.removeEventListener('abort', abort)
^
TypeError: this.removeEventListener is not a function
at EventEmitter.abort (/node_modules/undici/lib/web/fetch/request.js:57:12)
at EventEmitter.emit (node:events:519:28)
at EventEmitter.emit (node:domain:488:12)
at AbortSignal.dispatchEvent (/node_modules/node-abort-controller/index.js:28:23)
at AbortController.abort (/node_modules/node-abort-controller/index.js:58:17)
at Timeout._onTimeout (/node_modules/@apollo/gateway/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts:130:16)
at listOnTimeout (node:internal/timers:573:17)
at processTimers (node:internal/timers:514:7)
```
Looking at the error source this appears to be related to a request timeout, which calls `node-abort-controller` polyfill which in turn tries to call the undici removeEventListener function which doesn't exist.
https://github.com/apollographql/federation/blob/38beaa2f753eaf31be74b272ea62f3c1948c854f/gateway-js/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts#L130
https://github.com/southpolesteve/node-abort-controller/blob/10e0cea66a069d9319f948d055621e1d37aea5db/index.js#L16
### Link to Reproduction
N/A
### Reproduction Steps
Service Dependencies:
```
"dependencies": {
"@apollo/gateway": "^2.8.3",
"@apollo/server": "^4.10.4",
"@aws-sdk/client-ssm": "^3.577.0",
"body-parser": "^1.20.2",
"cors": "^2.8.5",
"dd-trace": "^5.0.0",
"express": "^4.19.2",
"fs": "0.0.1-security",
"global-agent": "^3.0.0",
"http-status-codes": "^2.1.4",
"jsonwebtoken": "^9.0.0",
"lodash": "^4.17.21",
"nodejs-base64": "^2.0.0",
"superagent": "^8.0.0",
"swagger-ui-express": "^5.0.0",
"undici": "^6.16.1",
"winston": "^3.8.1",
"ws": ">=6.2.2",
"yamljs": "^0.3.0"
},
```
Gateway init:
```
import { fetch } from 'undici'
const gateway = new ApolloGateway({
fetcher: fetch,
logger: appLogger,
});
```
Node version: 18
Contributor guide
Research direction
Start with gateway-js/src/supergraphManagers/UplinkSupergraphManager/loadSupergraphSdlFromStorage.ts around the timeout at line 130, then compare the abort handling in undici/lib/web/fetch/request.js with node-abort-controller/index.js. Reproduce the Node 18 timeout path if possible; done means the outage-related abort no longer crashes the gateway.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- graphql, node.js, typescript
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100