elastic / elastic/elastic-agent

[Flaky Test] github.com/elastic/elastic-agent/testing/integration/ess TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version

Open
#16,445 0 comments 0 reactions 0 assignees View on GitHub
flaky-test
Dominant language
Go
Stars
276
Forks
266
Avg merge
1d 23h
Merged PRs (30d)
312

Description

## Flaky Test

* **Test Name:** TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version
* **Scope:** github.com/elastic/elastic-agent/testing/integration/ess
* **File:** N/A
* **Location:** N/A
* **Buildkite Link:** https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/c821c4f1-f58e-8ecf-87f7-2218fdb7cd37
* **Flaky Instances:** 1
* **Latest Occurrence:** 2026-08-31T17:30:22.329Z

### Details

```json
{
"id": "c821c4f1-f58e-8ecf-87f7-2218fdb7cd37",
"web_url": "https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/c821c4f1-f58e-8ecf-87f7-2218fdb7cd37",
"scope": "github.com/elastic/elastic-agent/testing/integration/ess",
"name": "TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version",
"location": null,
"file_name": null,
"instances": 1,
"latest_occurrence_at": "2026-08-31T17:30:22.329Z",
"most_recent_instance_at": "2026-08-31T17:30:22.329Z",
"last_resolved_at": null,
"ownership_team_ids": [],
"failure_examples_count": 1
}
```

### Failure Examples

**Example 1:**
**Run:** https://api.buildkite.com/v2/analytics/organizations/elastic/suites/elastic-agent-ci/runs/3176f376-f507-8b2b-885e-7f38d00f5ca9
**Time:** 2026-08-31T15:28:25.052Z
**Stacktrace:**

```
=== RUN TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version
fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip
fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version1498174328
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version1498174328
fixture.go:1258: Components were not modified from the fetched artifact
repackage.go:49: originalPackageName: "elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip" newPackageFileName: "elastic-agent-9.6.0+build20260831170248-SNAPSHOT-windows-x86_64.zip"
repackage.go:61: file "elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip" is a .zip package
repackage.go:282: read old manifest: &{apiObject:{Version:co.elastic.agent/v1 Kind:PackageManifest} Package:{Version:9.6.0 Snapshot:true Hash:fbd7b99c62a725e1bcf6804cf26d5dfbb09a022b Fips:false VersionedHome:data/elastic-agent-fbd7b9 PathMappings:[map[data/elastic-agent-fbd7b9:data/elastic-agent-9.6.0-SNAPSHOT-fbd7b9 manifest.yaml:data/elastic-agent-9.6.0-SNAPSHOT-fbd7b9/manifest.yaml]] Flavors:map[basic:[elastic-otel-collector endpoint-security pf-host-agent] servers:[elastic-otel-collector endpoint-security pf-host-agent cloudbeat apm-server fleet-server pf-elastic-symbolizer pf-elastic-collector]]}}
repackage.go:288: generated new manifest:
version: co.elastic.agent/v1
kind: PackageManifest
package:
version: 9.6.0+build20260831170248
snapshot: true
hash: fbd7b99c62a725e1bcf6804cf26d5dfbb09a022b
versioned-home: data/elastic-agent-fbd7b9
path-mappings:
- data/elastic-agent-fbd7b9: data/elastic-agent-9.6.0+build20260831170248-SNAPSHOT-fbd7b9
manifest.yaml: data/elastic-agent-9.6.0+build20260831170248-SNAPSHOT-fbd7b9/manifest.yaml
repackage.go:247: writing new package version: "9.6.0+build20260831170248"
repackage.go:247: writing new package version: "9.6.0+build20260831170248"
upgrade_rollback_test.go:742: Performing initial install of version "9.6.0-SNAPSHOT" and first upgrade to version "9.6.0+build20260831170248-SNAPSHOT"
fixture.go:438: Extracting artifact elastic-agent-9.6.0+build20260831170248-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version2308389575
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0+build20260831170248-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version2308389575
fixture.go:1258: Components were not modified from the fetched artifact
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version1498174328\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version2308389575\elastic-agent-9.6.0+build20260831170248-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrader.go:245: installation of Elastic Agent will use --unprivileged as both start and end version support --unprivileged mode
upgrader.go:294: Installing version "9.6.0-SNAPSHOT"
fixture_install.go:201: [test TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version] Inside fixture install function
fixture_install.go:205:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/pkg/testing/fixture_install.go:205
C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/pkg/testing/fixture_install.go:193
C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/testing/upgradetest/upgrader.go:296
C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/testing/integration/ess/upgrade_rollback_test.go:747
Error: Should be empty, but was [{PID:3428, PPID: 1080, Cwd: , Exe: , Cmdline: C:\Program Files\Elastic\Agent\elastic-agent.exe, Args: [C:\Program Files\Elastic\Agent\elastic-agent.exe]}]
Test: TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version
Messages: there should be no running agent at beginning of Install()
fixture_install.go:229: [test TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version] Inside fixture installNoPkgManager function
fixture_install.go:260: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version1498174328\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged --checkin-on-state-change]
fixture.go:1389: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-azure-1788192111345676540\elastic\elastic-agent-extended-testing\build\diagnostics\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version-2026-08-31T17-03-26Z-ProcessDump-install.json
upgrade_rollback_test.go:681:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/testing/integration/ess/upgrade_rollback_test.go:681
C:/buildkite-agent/builds/bk-agent-prod-azure-1788192111345676540/elastic/elastic-agent-extended-testing/testing/integration/ess/upgrade_rollback_test.go:750
Error: Received unexpected error:
failed to install start agent (err: error running agent install command: exit status 1) [output: Unprivileged installation mode enabled.
Installing in non-interactive mode.

[ ] Uninstalling current Elastic Agent [0s]
[ ] Stopping service [0s]

[ ] Failed to issue stop service [0s]

[ ] Failed to uninstall agent [0s] Error uninstalling. Printing logs
2026-08-31T17:03:26.182Z DEBUG [uninstall] Loaded configuration from C:\Program Files\Elastic\Agent\elastic-agent.yml
2026-08-31T17:03:26.182Z DEBUG [uninstall] Merged configuration from C:\Program Files\Elastic\Agent\elastic-agent.yml into result
2026-08-31T17:03:26.182Z DEBUG [uninstall] Merged all configuration files from [C:\Program Files\Elastic\Agent\elastic-agent.yml], no external input files
Error: error uninstalling agent: failed to stop service (Elastic Agent): The service cannot accept control messages at this time.
For help, please see our troubleshooting guide at https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems

[ ] Uninstall failed [0s] Error: failed to uninstall elastic-agent: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/docs/troubleshoot/ingest/fleet/common-problems
]
Test: TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version
Messages: upgrade should not fail
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version2308389575" preserved for investigation/debugging
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestCleanupRollbacks-agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version1498174328" preserved for investigation/debugging
--- FAIL: TestCleanupRollbacks/agent_should_clear_expired_rollback_after_upgrading_to_a_repackaged_version (37.58s)
```

Contributor guide

Open the contributing guide

Research direction

Start with testing/integration/ess/upgrade_rollback_test.go, especially TestCleanupRollbacks and the install path reported at fixture_install.go:205. Review the Buildkite failure and process dump to understand why the Windows agent is still running when installation begins. Done means the test no longer fails intermittently during service stopping or initial installation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.