elastic / elastic/apm-agent-nodejs
Adaptive Sampling
- Dominant language
- JavaScript
- Stars
- 594
- Forks
- 244
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 16
Description
**Is your feature request related to a problem? Please describe.**
It is way too easy to completely nuke your API by deploying the default APM configuration. Specifically with how the `transactionSampleRate` option works. By default, it will trace every transaction which will completely overwhelm many API servers and also inundate the Elastic APM server and related elastic database with requests/data. We were/are looking for a New Relic alternative and their [agent](https://github.com/newrelic/node-newrelic/blob/main/lib/adaptive-sampler.js) uses [adaptive sampling](https://docs.newrelic.com/docs/understand-dependencies/distributed-tracing/get-started/how-new-relic-distributed-tracing-works) by default with a max number of transactions traced per minute.
**Describe the solution you'd like**
Implement adaptive sampling based on real-time traffic. It should look at some predefined period of time, see how many transactions/requests are performed in that window, and then for the next window of time, it should dynamically set the % of transactions to sample based on a target number for that period. This approach would also automatically adjust and scale well for APIs that see cyclical traffic on a daily basis. It would also auto-adjust as your API (hopefully!) gets more traffic over time.
This could be a global setting but perhaps there is also some intelligence on a per path basis as well. For example, if endpoint `/abc` gets 200 req/sec and endpoint `/123` only gets 2 req/sec it is very unlikely `/123` will be sampled even though this would be very useful to have and would be of minimal additional overhead. Not to compare to New Relic too much, but somehow they make these work as even very low traffic endpoints get at least a few traces.
I think this could be implemented by passing a configurable object to the existing `transactionSampleRate`.
Something like the following, but I haven't put a ton of thought into the API configuration options...
```
transactionSampleRate: {
period: 60 //Seconds
samples: 10 //target,samples,numberOfSamples,etc.
}
```
I think it would also make sense for this to be the default implementation.
**Describe alternatives you've considered**
I am not aware of any alternatives.
**Additional context**
None at this time.
Contributor guide
Assessment
This issue has not been assessed yet.