apollographql / apollographql/apollo-server

Memory leak in ApolloServerPluginUsageReporting plugin

Open
#7,639 11 comments 0 reactions 1 assignee Claimed by @trevor-scheer View on GitHub
Dominant language
TypeScript
Stars
14k
Forks
2k
Avg merge
2d 14h
Merged PRs (30d)
2

Description

### Issue Description

We've been running Apollo server for a while in a couple of APIs, and we have always noticed a memory leak in both, which appeared to be linearly proportional to the number of requests handled by each API.

While investigating the memory leak, v8 heap snapshots where taken from the running servers at two different timestamps, with a distance of 6 hours. The latter heap snapshot was compared to the previous one in order to track what new objects are in the JS heap that where not 6 hours before, and there are thousands of new retained `Request`-like objects that reference the "usage-reporting.api.apollographql.com" host, and hundreds of `TLSSocket` new objects that reference this same host.

### Some objects that are leaking in the JS memory:

Request-like object

```
body::Object@13534193
cache::"default"@729πŸ—–
client::Object@13537293
credentials::"same-origin"@54437πŸ—–
cryptoGraphicsNonceMetadata::""@77πŸ—–
destination::""@77πŸ—–
done::system / Oddball@73πŸ—–
headersList::HeadersList@13537317
historyNavigation::system / Oddball@75πŸ—–
initiator::""@77πŸ—–
integrity::""@77πŸ—–
keepalive::system / Oddball@75πŸ—–
localURLsOnly::system / Oddball@75πŸ—–
map::system / Map@130579
method::"POST"@49427πŸ—–
mode::"cors"@84517πŸ—–
origin::system / Oddball@67πŸ—–
parserMetadata::""@77πŸ—–
policyContainer::Object@13537295
preventNoCacheCacheControlHeaderModification::system / Oddball@75πŸ—–
priority::system / Oddball@71πŸ—–
properties::system / PropertyArray@13537319
redirect::"follow"@53093πŸ—–
referrer::"no-referrer"@85507πŸ—–
referrerPolicy::system / Oddball@67πŸ—–
reloadNavigation::system / Oddball@75πŸ—–
replacesClientId::""@77πŸ—–
reservedClient::system / Oddball@71πŸ—–
responseTainting::"basic"@102749πŸ—–
serviceWorkers::"none"@519πŸ—–
taintedOrigin::system / Oddball@75πŸ—–
timingAllowFailed::system / Oddball@75πŸ—–
unsafeRequest::system / Oddball@75πŸ—–
url::URL@13537301
::URLContext@13538143
fragment::system / Oddball@71πŸ—–
host::"usage-reporting.api.apollographql.com"@13538145πŸ—–
map::system / Map@135759
path::Array@13538147
port::system / Oddball@71πŸ—–
query::system / Oddball@71πŸ—–
scheme::"https:"@6945πŸ—–
username::""@77πŸ—–
__proto__::Object@135757
::URLSearchParams@13538149
map::system / Map@135741
__proto__::URL@135739πŸ—–
urlList::Array@13537299
useCORSPreflightFlag::system / Oddball@75πŸ—–
useCredentials::system / Oddball@75πŸ—–
userActivation::system / Oddball@75πŸ—–
window::"no-window"@87117πŸ—–
__proto__
```

TLSSocket object

```
::system / Oddball@75πŸ—–
::Client@131765
::Object@13536139
::InformationalError@13536143
::system / Oddball@71πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@75πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@75πŸ—–
::system / Oddball@73πŸ—–
::system / Oddball@67πŸ—–
::system / Oddball@73πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@75πŸ—–
::system / Oddball@71πŸ—–
::system / Oddball@73πŸ—–
::system / Oddball@75πŸ—–
_SNICallback::system / Oddball@71πŸ—–
_closeAfterHandlingError::system / Oddball@75πŸ—–
_controlReleased::system / Oddball@73πŸ—–
_events::Object@13536133
_hadError::system / Oddball@75πŸ—–
_host::"usage-reporting.api.apollographql.com"@131813πŸ—–
_maxListeners::system / Oddball@67πŸ—–
_newSessionPending::system / Oddball@75πŸ—–
_parent::system / Oddball@71πŸ—–
_peername::Object@13536141
_pendingData::system / Oddball@71πŸ—–
_pendingEncoding::""@77πŸ—–
_readableState::ReadableState@13536135
_rejectUnauthorized::system / Oddball@73πŸ—–
_requestCert::system / Oddball@73πŸ—–
_secureEstablished::system / Oddball@73πŸ—–
_securePending::system / Oddball@75πŸ—–
_server::system / Oddball@71πŸ—–
_sockname::system / Oddball@71πŸ—–
_tlsOptions::Object@13536129
_writableState::WritableState@13536137
allowHalfOpen::system / Oddball@75πŸ—–
alpnProtocol::system / Oddball@75πŸ—–
authorizationError::system / Oddball@71πŸ—–
authorized::system / Oddball@73πŸ—–
connecting::system / Oddball@75πŸ—–
domain::system / Oddball@71πŸ—–
encrypted::system / Oddball@73πŸ—–
map::system / Map@130053
properties::system / PropertyArray@13536145
secureConnecting::system / Oddball@75πŸ—–
server::system / Oddball@67πŸ—–
servername::"usage-reporting.api.apollographql.com"@13536131πŸ—–
ssl::system / Oddball@71πŸ—–
__proto__::Socket@147607πŸ—–
```

Here is a chart showing the memory usage of the last two days for one of the APIs:

![Screenshot 2023-07-12 at 09 17 21](https://github.com/apollographql/apollo-server/assets/45515538/683cae90-0b57-4312-a53c-29ed54612800)

The first left half of the chart (the first day) the Apollo server was running with the `ApolloServerPluginUsageReporting` enabled, and the memory kept increasing linearly, and the last half (the second day), exactly the same code was running but passing the `ApolloServerPluginUsageReportingDisabled` to the plugins, so that the usage reporting is disabled. In this last case no memory was being leaked.

We are using `@apollo/server` with version `4.3.0`

### Link to Reproduction

https://github.com/GabrielMusatMestre/apollo-server-memory-leak-repro

### Reproduction Steps

Steps are described in the README.md of the reproduction repo.

This is not a reliable reproduction, as the memory leak might start being noticeable by running the server under heavy load for hours or days, and it needs a properly configured `APOLLO_KEY` and `APOLLO_GRAPH_REF` that will actually publish usage reports to Apollo.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.