[META] Mobile span timestamp correction is not working in some cases
- Dominant language
- Gherkin
- Stars
- 427
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The [approach we currently have for correcting the timestamp for mobile agents](https://github.com/elastic/apm-server/blob/d3297ebcbee15db54fbdcd9885ef187c75b83ec8/dev_docs/otel.md#time-synchronisation) is not solving the problems in case there is network latency or the APM server being overloaded.
Let's consider the following example:

The above mentioned time correction approach assumes that the network time is negligible. So, with the mobile agents setting the export timestamp, the APM server assumes it's the same timestamp as the receive timestamp on the APM server time.
> This has the side effect of increasing timestamps by client/server network latency, which is typically an acceptable tradeoff.
This tradeoff becomes a problem if the network latency is not consistent for different spans from the same trace.
In the above screenshot the first two spans have been sent with a short latency, but the third span has been sent with a way bigger network latency (or the APM server queue was filled at that time). The result is that the third span has a relatively negative timestamp compared to its parent which skews the trace view.
## Solution Idea
Instead of correcting the timestamp at the APM server we are thinking about fixing the timestamp within the Mobile agents by using an NTP service that would allow us to set correct timestamps right away. This wouldn't require fixing the time at the server anymore.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.