elastic / elastic/apm-data

OTEL: Exceptions not always shown human-readable in Kibana because of parsing result

Open
#401 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
12
Forks
31
Avg merge
8h 55m
Merged PRs (30d)
2

Description

Hi, this might be simply parsing issue or Kibana display issue, I do not know. We are using OpenTelemetry to send logs and traces via APM Server into Elastic, our version is 8.16.0.

https://github.com/elastic/apm-data/blob/1c7a133d0671fded4ba35205ef9658c65af087bd/input/otlp/exceptions.go#L90-L95

This code either fills `Error.Exception.Stacktrace` (object with various attributes) or `Error.StackTrace` (string). Kibana is unfortunately not able to display usable (human-readable) stack traces in Logs Explorer if they come parsed (see screenshots for comparison), according to code the Kibana has some code able to render parsed exceptions, but I have not seen it working neither in APM Errors, Traces nor in Logs Explorer.

For the unparsed message there would have to be a more tolerant parser, because the stack trace looks like the following:

```
com.google.api.client.googleapis.json.GoogleJsonResponseException: 501 Not Implemented
POST https://analytics.googleapis.com/upload/analytics/v3/management/accounts/54242820/webproperties/UA-54242820-11/customDataSources/De9KaW0jR8ufoMGGZv0epQ/uploads?uploadType=resumable
{
"code": 501,
"errors": [
{
"domain": "global",
"message": "Operation is not implemented, or supported, or enabled.",
"reason": "notImplemented"
}
],
"message": "Operation is not implemented, or supported, or enabled.",
"status": "UNIMPLEMENTED"
}
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:583)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:506)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:616)

```

Parsed error with no human-readable stack trace in Logs Explorer:
![Parsed error with no human-readable stack trace](https://github.com/user-attachments/assets/2c5c4a95-e06e-437f-9f35-a84a11905b3d)

Unparsed error (there is a POST request logged as part of stack trace from Google library):
![Unparsed error](https://github.com/user-attachments/assets/9f3ca26f-6ea2-45e0-8d0f-d6238939165e)

Anyway, we see no exceptions in APM Errors, though.

No stack trace in APM Errors display:
![No stack trace](https://github.com/user-attachments/assets/9070b00f-0d3a-4e1a-b496-0a243de95418)

The trace document contains parsed exception:
![Trace document](https://github.com/user-attachments/assets/b6f60ba7-5196-4eaa-8542-6fc8ce9afaf2)

Kibana is trying to detect exceptions even in the parsed state, but the check is actually useless and looks like a Kibana bug - the `error.exception.stacktrace` does not have any direct value for parsed exceptions, they are stored in parallel-structure sub-attributes `error.exception.stacktrace.*`:

https://github.com/elastic/kibana/blob/a41017e74e4f495c6892fbe023cff0045839bd9b/packages/kbn-discover-utils/src/utils/get_stack_trace_fields.ts#L18-L20

We used APM Java Agent until now and this always worked, so to me this looks more like an issue in conversion from OpenTelemetry messages into Elastic documents rather than Kibana display issue.

Is any more information needed? Should I create also Kibana issue? Feel free to suggest next steps.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.