Various ambiguous APM server errors in the logs are not useful
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description

We see in some situations (e.g. when APM server is all queued up), tens of thousands of ambiguous/brief error messages (e.g. `data validation error`, `internal error`, `EOF`) that are not useful for troubleshooting. Here are examples of various ambiguous errors in APM Server logs and suggested improvements:
**data validation error**
- Can we include what part of the document structure is invalid? Or instruct the user to enable debugging to find out?
- Can we include some "identification" information so users can quickly determine the origin of these rejected events?
**internal error**
- Can we indicate what exactly is the error here?
**unexpected EOF**
- Can we indicate what is wrong here or if it's a side effect of some other error?
**failed to publish events: Post "/_bulk": context deadline exceeded (Client.Timeout exceeded while awaiting headers)**
- Can we explain which "client" timeout it is and reference the documentation for information on how one can adjust this (or should they)? The APM agent's?
**failed to publish events: Post "/_bulk": EOF**
- Can we indicate what is wrong here or if it's a side effect of some other error?
**404 page not found**
- What page are we referring to? What request are we trying to make?
**sending request to kibana failed**
- What request are we trying to make? What is the impact if such request fails?
**request timed out**
- What request are we trying to make (are we referring to indexing request to Elasticsearch)? What is the impact if such request fails?
**request size exceeded**
- Can we include the size of the problem event and the current "size limit" in the message?
- Can we include some "identification" information so users can quickly determine the origin of these rejected events?
It will be a better experience to provide details on what APM Server is complaining about :) For errors that can repeat tens of thousands of times, we might want to suppress some of these repeated errors that tend to drown the APM Server log file (e.g., if it's the same error, emit 1 error type every N minutes with the # of occurrences).
Contributor guide
Research direction
The issue names several ambiguous APM Server log messages but no files, tests, or entry points. Start by locating where each listed message is emitted and reviewing the related logging and error-handling tests. Done means the affected errors provide actionable context and any repeated-error suppression behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100