elastic / elastic/apm-server

Duplicate events in logs-apm.error datastream when log sending is enabled on agents

Open
#13,743 1 comment 0 reactions 0 assignees View on GitHub
bug operational
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

**APM Server version** (`apm-server version`): All versions supporting log ingestion

**Description of the problem including expected versus actual behavior**:
The current datastream routing code uses [event.Type()](https://github.com/elastic/apm-data/blob/cee5ac3fc5f2ee66f156133992882cae758e2b66/model/modelprocessor/datastream.go#L70) which is inferred based on the below logic:

```go
func (a *APMEvent) Type() APMEventType {
switch {
case a.Metricset != nil:
return MetricEventType
case a.Error != nil:
return ErrorEventType
case a.Log != nil || a.Event.GetKind() == "event":
return LogEventType
case a.Span.GetType() != "":
return SpanEventType
case a.Transaction.GetType() != "":
return TransactionEventType
}
return UndefinedEventType
}
```

([ref](https://github.com/elastic/apm-data/blob/cee5ac3fc5f2ee66f156133992882cae758e2b66/model/modelpb/apmevent.pb.type.go#L63-L77))
A log event with stacktrace, error type, or error message, is parsed into the `APMEvent#Error`. Which would route them to error datastream (`logs-apm.error.*`).
OTOH, for an error event captured by the agent with error type, error message, and stacktrace as `Exception#Stacktrace` will also be parsed into `APMEvent#Error`.

Meaning both the above events would be treated as an error type rather than log type. This leads to the issue when an agent is configured to capture logs. In this case, both the error event and the log will be sent to APM-Server and would end up in the same datastream causing duplicate error events. In addition, the grouping key for these would be difference since the grouping key for an error event would used the parsed stack trace ([ref](https://github.com/elastic/apm-data/blob/cee5ac3fc5f2ee66f156133992882cae758e2b66/model/modelprocessor/groupingkey.go#L47-L75)).

**Steps to reproduce**:

Send an error event and a log event for the same error to APM-Server and observe both being indexed in the same, `logs-apm.error*` datastream.

Please include a *minimal* but *complete* recreation of the problem,

**Provide logs (if relevant)**: N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.