elastic / elastic/apm

[META] Mobile span timestamp correction is not working in some cases

Open
#710 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

## Problem

The [approach we currently have for correcting the timestamp for mobile agents](https://github.com/elastic/apm-server/blob/d3297ebcbee15db54fbdcd9885ef187c75b83ec8/dev_docs/otel.md#time-synchronisation) is not solving the problems in case there is network latency or the APM server being overloaded.

Let's consider the following example:
![image](https://user-images.githubusercontent.com/866830/197794188-f0e45a00-423a-401c-b1d4-af8fcd8c3c40.png)

The above mentioned time correction approach assumes that the network time is negligible. So, with the mobile agents setting the export timestamp, the APM server assumes it's the same timestamp as the receive timestamp on the APM server time.

> This has the side effect of increasing timestamps by client/server network latency, which is typically an acceptable tradeoff.

This tradeoff becomes a problem if the network latency is not consistent for different spans from the same trace.

In the above screenshot the first two spans have been sent with a short latency, but the third span has been sent with a way bigger network latency (or the APM server queue was filled at that time). The result is that the third span has a relatively negative timestamp compared to its parent which skews the trace view.

## Solution Idea
Instead of correcting the timestamp at the APM server we are thinking about fixing the timestamp within the Mobile agents by using an NTP service that would allow us to set correct timestamps right away. This wouldn't require fixing the time at the server anymore.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.