elastic / elastic/elastic-agent
[Flaky Test] github.com/elastic/elastic-agent/testing/integration/ess TestFleetManagedUpgradeUnprivileged
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
## Flaky Test
* **Test Name:** TestFleetManagedUpgradeUnprivileged
* **Scope:** github.com/elastic/elastic-agent/testing/integration/ess
* **File:** N/A
* **Location:** N/A
* **Buildkite Link:** https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/ef54688e-de25-8264-9889-5d5325ec3df6
* **Flaky Instances:** 1
* **Latest Occurrence:** 2026-08-26T23:00:33.859Z
### Details
```json
{
"id": "ef54688e-de25-8264-9889-5d5325ec3df6",
"web_url": "https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/ef54688e-de25-8264-9889-5d5325ec3df6",
"scope": "github.com/elastic/elastic-agent/testing/integration/ess",
"name": "TestFleetManagedUpgradeUnprivileged",
"location": null,
"file_name": null,
"instances": 1,
"latest_occurrence_at": "2026-08-26T23:00:33.859Z",
"most_recent_instance_at": "2026-08-26T23:00:33.859Z",
"last_resolved_at": null,
"ownership_team_ids": [],
"failure_examples_count": 1
}
```
### Failure Examples
**Example 1:**
**Run:** https://api.buildkite.com/v2/analytics/organizations/elastic/suites/elastic-agent-ci/runs/2019d559-987c-816b-940a-201d7d6b6ceb
**Time:** 2026-08-26T20:56:54.145Z
**Stacktrace:**
```
=== RUN TestFleetManagedUpgradeUnprivileged
fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip
fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771
fixture.go:1258: Components were not modified from the fetched artifact
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip
fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604
fixture.go:1258: Components were not modified from the fetched artifact
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_fleet_test.go:114: Testing Elastic Agent upgrade from 9.6.0-SNAPSHOT to 9.6.0-SNAPSHOT with Fleet...
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_fleet_test.go:412: Creating Agent policy...
upgrade_fleet_test.go:419: Creating Agent enrollment API key...
upgrade_fleet_test.go:428: Getting default Fleet Server URL...
upgrade_fleet_test.go:435: Installing Elastic Agent (unprivileged: true)...
fixture_install.go:201: [test TestFleetManagedUpgradeUnprivileged] Inside fixture install function
fixture_install.go:229: [test TestFleetManagedUpgradeUnprivileged] Inside fixture installNoPkgManager function
fixture_install.go:260: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged --url https://4c574a8d2ad35ef68edd7ca4cf765e2d.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token czNwRVFLQUJqS0hoTzVKS3Bnakg6Vm1MTXQ2VnRzUDZWNzBmMm1uV0tMQQ== --checkin-on-state-change]
upgrade_fleet_test.go:450: install start agent output:
Unprivileged installation mode enabled.
Installing in non-interactive mode.
[ ] finding group: "elastic-agent"(S-1-5-21-74956108-3170057274-2946573734-1007): [0s]
[ ] finding user with forceCreate set to true: "elastic-agent-user"(S-1-5-21-74956108-3170057274-2946573734-1008): [0s]
[ ] Copying install files [0s]
[ ] Copying install files [0s]
[ ==] Copying install files [0s]
[ ===] Copying install files [0s]
[ ===] Copying install files [0s]
[=== ] Copying install files [0s]
[=== ] Copying install files [0s]
[== ] Copying install files [0s]
[ ] Copying install files [0s]
[ =] Copying install files [0s]
[ ==] Copying install files [1s]
[ ===] Copying install files [1s]
[====] Copying install files [1s]
[=== ] Copying install files [1s]
[== ] Copying install files [1s]
[= ] Copying install files [1s]
[ ] Copying install files [1s]
[ =] Copying install files [1s]
[ ==] Copying install files [1s]
[ ===] Copying install files [1s]
[====] Copying install files [2s]
[=== ] Copying install files [2s]
[== ] Copying install files [2s]
[= ] Copying install files [2s]
[= ] Successfully copied files [2s]
[= ] Successfully copied files [2s]
[ =] Successfully copied files [2s]
[ =] Installing service [2s]
[ ==] Installing service [2s]
[ ===] Installing service [2s]
[====] Installing service [2s]
[====] Installed service [2s]
[====] Starting Service [2s]
[====] Starting Service [2s]
[=== ] Service Started [2s] Elastic Agent successfully installed, starting enrollment.
[=== ] Enrolling Elastic Agent with Fleet [2s]
[=== ] Waiting For Enroll... [2s]
[== ] Waiting For Enroll... [3s] {"log.level":"info","@timestamp":"2026-08-26T22:50:41.131Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":87},"message":"Starting enrollment to URL: https://4c574a8d2ad35ef68edd7ca4cf765e2d.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
[= ] Waiting For Enroll... [3s]
[ ] Waiting For Enroll... [3s]
[ =] Waiting For Enroll... [3s]
[ ==] Waiting For Enroll... [3s]
[ ===] Waiting For Enroll... [3s]
[ ===] Waiting For Enroll... [3s]
[====] Waiting For Enroll... [3s]
[=== ] Waiting For Enroll... [3s]
[== ] Waiting For Enroll... [3s]
[ ] Waiting For Enroll... [4s]
[ ] Waiting For Enroll... [4s]
[ ==] Waiting For Enroll... [4s]
[ ===] Waiting For Enroll... [4s]
[ ===] Waiting For Enroll... [4s]
[=== ] Waiting For Enroll... [4s] {"log.level":"info","@timestamp":"2026-08-26T22:50:42.640Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":417},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-08-26T22:50:42.644Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":206},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=== ] Enroll Completed [4s]
[=== ] Done [4s]
[=== ] Done [4s]
Elastic Agent has been successfully installed.
upgrade_fleet_test.go:464: Waiting for Agent to be correct version and healthy...
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrade_fleet_test.go:474: Agent ID: "e66693fc-7041-4e4e-a3d1-ac43d60a8c09"
upgrade_fleet_test.go:476: Waiting for enrolled Agent status to be online...
upgrade_fleet_test.go:489: Upgrading from version "9.6.0-SNAPSHOT-540fc31f60c1cc58c5b448388ed70e69af69ac16" to version "9.6.0-SNAPSHOT-0fc03a8a0c522f168b0d8f8f79b35a75873b66f0"...
upgrade_fleet_test.go:497: Waiting from upgrade details to show up in Fleet
upgrade_fleet_test.go:513: Waiting for upgrade watcher to start...
upgrade_fleet_test.go:518: Upgrade watcher started
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
upgrade_fleet_test.go:364:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:364
C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:117
C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:57
Error: Received unexpected error:
waiting for agent to be healthy and version 9.6.0-SNAPSHOT: failed waiting for healthy agent and version (context deadline exceeded): commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
Test: TestFleetManagedUpgradeUnprivileged
fixture_install.go:325: [test TestFleetManagedUpgradeUnprivileged] Inside fixture cleanup function
fixture_install.go:341: collecting diagnostics; test failed
fixture_install.go:833: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1787780316924129446\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeUnprivileged-2026-08-26T22-53-37Z-diagnostics.zip]
fixture_install.go:784: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
fixture.go:1389: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1787780316924129446\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeUnprivileged-2026-08-26T22-53-37Z-ProcessDump-cleanup.json
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeUnprivileged360939604" preserved for investigation/debugging
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeUnprivileged215907771" preserved for investigation/debugging
--- FAIL: TestFleetManagedUpgradeUnprivileged (201.69s)
```
Contributor guide
Research direction
Start with TestFleetManagedUpgradeUnprivileged in the github.com/elastic/elastic-agent/testing/integration/ess scope and review the failure output from the linked Buildkite run. Trace the upgrade flow through upgrade_fleet_test.go, fixture.go, fixture_install.go, and upgrader.go, focusing on the reported health and commit mismatches. Done means the test no longer flakes and reliably reaches the expected upgraded version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100