elastic / elastic/elastic-agent

Add system tests for APM against combination of opentelemetry-kube-stack and otel components

Open
#14,957 0 comments 1 reaction 2 assignees Claimed by @sboomsma View on GitHub
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

## Summary

Elastic Agent (EA) has a regression where EA 9.3.2 does not produce metrics for service_destination correctly. This caused Dependencies to not show in Kibana APM Applications -> Dependencies for a customer using OTEL mapping mode and opentelemetry-kube-stack.

## Proposed action

- Port a comprehensive set of system tests from apm-server to the elastic/elastic-agent repo that cover the full intakev2/OTLP -> Elasticsearch document flow for the APM use case, including all aggregated metrics.
- An alternative to the above is to reuse any existing managed OTLP e2e tests
- Ensure these tests run with combinations that mirror Kibana onboarding instructions: for a given stack release version, e.g. 9.3.2, test the collector config in kube-stack Helm chart values in combination with the corresponding EA pinned component versions
- Add CI e2e test jobs (or extend existing e2e CI)

## Context & Rationale

- A customer case demonstrates that traces contain destination/db/span metadata but the generated metrics stream `metrics-service_destination.*` returned zero documents in 9.3.2 with opentelemetry-kube-stack values and EDOT configuration. Service map works but Dependencies view in APM is empty.
- The recent investigation indicates EA 9.3.2 used a broken combination of elasticapmprocessor and elasticapmconnector which removed/failed to generate the required field for service_destination metrics. Upgrading to 9.3.3 fixes the issue.
- This highlights lack of test coverage for the APM/EDOT end-to-end flow. Having system tests would catch mismatched component-version regressions earlier and reduce customer impact and investigation time.

## Acceptance Criteria

- A suite of system tests from apm-server is ported to elastic-agent that validate trace and aggregated metrics documents.
- Tests run in CI for selected kube-stack + EA combinations (including the combinations used in Kibana onboarding instructions) and fail the build on regressions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.