elastic / elastic/integrations

Investigate Flaky System Test Failures: "Could not find the expected hits" Across Multiple Integrations

Open
#16,310 6 comments 0 reactions 0 assignees View on GitHub
Team:Ecosystem
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

We have observed recurring flaky test failures in daily builds (and in some Pull Requests too), specifically with the error message:

> "test case failed: could not find the expected hits in ."

These failures are tracked under the [flaky-test label](https://github.com/elastic/integrations/issues?q=test%20case%20failed%3A%20could%20not%20find%20the%20expected%20hits%20label%3Aflaky-test)

The root cause is unclear and may require deeper analysis of the build environment, service access, and test orchestration.

Please investigate the underlying cause of these flaky test failures (environment setup, Elastic Agent logs accessing services,...).

## Observations

- **Tests pass locally**: The issue does not reproduce when running tests on local environments.
- **Resource constraints**: Initial suspicion was resource bottlenecks (CPU/memory) in the build environment, but GCP metrics do not show clear evidence of this.

metrics for a failing CI step

- **Affected packages**: Many failing tests are for packages that create Terraform resources in system tests (e.g., aws, sentinel_one_cloud_funnel, trellix_edr_cloud, sublime_security, cisco_umbrella). However, some affected packages do not use Terraform in system tests (e.g., netskope, couchbase, google_workspace, network_traffic).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.