modelindexer doesn't ever respond with 503 to APM agents
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
## Description
While working on the adaptive sampling project during OnWeek, the APM server never returned a 503 back to the APM agents even when the number of transactions per second was quite elevated (300,000 transactions per minute or 5000 transactions/s).
The current `modelindexer` architecture differs from the previously used libbeat queues, and won't cause the APM Server HTTP server to respond back to the APM agents with a 503, since there are no "full queues" concept.
When all the bulk indexers are busy flushing, the indexer may have no available bulk indexer to retrieve from the available buffered channel: https://github.com/elastic/apm-server/blob/bf75ef5e0fbebc6a65f8095231175ff95eb26cd7/model/modelindexer/indexer.go#L232-L236
Which means that the APM Server will continue accepting events from the APM agents and the current goroutine will most likely be put in a sleeping state since it's receiving from a channel. This isn't necessarily bad, but it may increase the memory consumption since there are no bounds on how many batches may be waiting for the bulk indexers to be released back after they've been flushed.
**EDIT**: The APM Server will return 503 when the context is cancelled when it can't process the ingest requests on time, but we may want to provide some back pressure before that happens.
## Possible solutions
Since we're going to be using the new `modelindexer` from `8.0.0` onwards, it may be good to explore some ways to limit time that we wait for a bulk indexer to become available, measure and quantify the impact that this has.
Contributor guide
Assessment
This issue has not been assessed yet.