elastic / elastic/elastic-agent
Add system tests for APM against combination of opentelemetry-kube-stack and otel components
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
## Summary
Elastic Agent (EA) has a regression where EA 9.3.2 does not produce metrics for service_destination correctly. This caused Dependencies to not show in Kibana APM Applications -> Dependencies for a customer using OTEL mapping mode and opentelemetry-kube-stack.
## Proposed action
- Port a comprehensive set of system tests from apm-server to the elastic/elastic-agent repo that cover the full intakev2/OTLP -> Elasticsearch document flow for the APM use case, including all aggregated metrics.
- An alternative to the above is to reuse any existing managed OTLP e2e tests
- Ensure these tests run with combinations that mirror Kibana onboarding instructions: for a given stack release version, e.g. 9.3.2, test the collector config in kube-stack Helm chart values in combination with the corresponding EA pinned component versions
- Add CI e2e test jobs (or extend existing e2e CI)
## Context & Rationale
- A customer case demonstrates that traces contain destination/db/span metadata but the generated metrics stream `metrics-service_destination.*` returned zero documents in 9.3.2 with opentelemetry-kube-stack values and EDOT configuration. Service map works but Dependencies view in APM is empty.
- The recent investigation indicates EA 9.3.2 used a broken combination of elasticapmprocessor and elasticapmconnector which removed/failed to generate the required field for service_destination metrics. Upgrading to 9.3.3 fixes the issue.
- This highlights lack of test coverage for the APM/EDOT end-to-end flow. Having system tests would catch mismatched component-version regressions earlier and reduce customer impact and investigation time.
## Acceptance Criteria
- A suite of system tests from apm-server is ported to elastic-agent that validate trace and aggregated metrics documents.
- Tests run in CI for selected kube-stack + EA combinations (including the combinations used in Kibana onboarding instructions) and fail the build on regressions.
Contributor guide
Assessment
This issue has not been assessed yet.