elastic / elastic/apm-server

modelindexer doesn't ever respond with 503 to APM agents

Open
#6,953 6 comments 0 reactions 0 assignees View on GitHub
discuss v8.0.0
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

## Description

While working on the adaptive sampling project during OnWeek, the APM server never returned a 503 back to the APM agents even when the number of transactions per second was quite elevated (300,000 transactions per minute or 5000 transactions/s).

The current `modelindexer` architecture differs from the previously used libbeat queues, and won't cause the APM Server HTTP server to respond back to the APM agents with a 503, since there are no "full queues" concept.

When all the bulk indexers are busy flushing, the indexer may have no available bulk indexer to retrieve from the available buffered channel: https://github.com/elastic/apm-server/blob/bf75ef5e0fbebc6a65f8095231175ff95eb26cd7/model/modelindexer/indexer.go#L232-L236

Which means that the APM Server will continue accepting events from the APM agents and the current goroutine will most likely be put in a sleeping state since it's receiving from a channel. This isn't necessarily bad, but it may increase the memory consumption since there are no bounds on how many batches may be waiting for the bulk indexers to be released back after they've been flushed.

**EDIT**: The APM Server will return 503 when the context is cancelled when it can't process the ingest requests on time, but we may want to provide some back pressure before that happens.

## Possible solutions

Since we're going to be using the new `modelindexer` from `8.0.0` onwards, it may be good to explore some ways to limit time that we wait for a bulk indexer to become available, measure and quantify the impact that this has.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.