elastic / elastic/elastic-agent

[Flaky Test] github.com/elastic/elastic-agent/testing/integration/ess TestFleetManagedUpgradeUnprivileged

Open
#16,385 0 comments 0 reactions 0 assignees View on GitHub
flaky-test
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

## Flaky Test

* **Test Name:** TestFleetManagedUpgradeUnprivileged
* **Scope:** github.com/elastic/elastic-agent/testing/integration/ess
* **File:** N/A
* **Location:** N/A
* **Buildkite Link:** https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/ef54688e-de25-8264-9889-5d5325ec3df6
* **Flaky Instances:** 1
* **Latest Occurrence:** 2026-08-26T23:00:33.859Z

### Details

```json
{
"id": "ef54688e-de25-8264-9889-5d5325ec3df6",
"web_url": "https://buildkite.com/organizations/elastic/analytics/suites/elastic-agent-ci/tests/ef54688e-de25-8264-9889-5d5325ec3df6",
"scope": "github.com/elastic/elastic-agent/testing/integration/ess",
"name": "TestFleetManagedUpgradeUnprivileged",
"location": null,
"file_name": null,
"instances": 1,
"latest_occurrence_at": "2026-08-26T23:00:33.859Z",
"most_recent_instance_at": "2026-08-26T23:00:33.859Z",
"last_resolved_at": null,
"ownership_team_ids": [],
"failure_examples_count": 1
}
```

### Failure Examples

**Example 1:**
**Run:** https://api.buildkite.com/v2/analytics/organizations/elastic/suites/elastic-agent-ci/runs/2019d559-987c-816b-940a-201d7d6b6ceb
**Time:** 2026-08-26T20:56:54.145Z
**Stacktrace:**

```
=== RUN TestFleetManagedUpgradeUnprivileged
fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip
fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771
fixture.go:1258: Components were not modified from the fetched artifact
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fetcher.go:103: Using existing artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip
fixture.go:438: Extracting artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604
fixture.go:456: Completed extraction of artifact elastic-agent-9.6.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604
fixture.go:1258: Components were not modified from the fetched artifact
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_fleet_test.go:114: Testing Elastic Agent upgrade from 9.6.0-SNAPSHOT to 9.6.0-SNAPSHOT with Fleet...
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
fixture.go:1034: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged360939604\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe version --binary-only --yaml]
upgrade_fleet_test.go:412: Creating Agent policy...
upgrade_fleet_test.go:419: Creating Agent enrollment API key...
upgrade_fleet_test.go:428: Getting default Fleet Server URL...
upgrade_fleet_test.go:435: Installing Elastic Agent (unprivileged: true)...
fixture_install.go:201: [test TestFleetManagedUpgradeUnprivileged] Inside fixture install function
fixture_install.go:229: [test TestFleetManagedUpgradeUnprivileged] Inside fixture installNoPkgManager function
fixture_install.go:260: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestFleetManagedUpgradeUnprivileged215907771\elastic-agent-9.6.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged --url https://4c574a8d2ad35ef68edd7ca4cf765e2d.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token czNwRVFLQUJqS0hoTzVKS3Bnakg6Vm1MTXQ2VnRzUDZWNzBmMm1uV0tMQQ== --checkin-on-state-change]
upgrade_fleet_test.go:450: install start agent output:
Unprivileged installation mode enabled.
Installing in non-interactive mode.

[ ] finding group: "elastic-agent"(S-1-5-21-74956108-3170057274-2946573734-1007): [0s]

[ ] finding user with forceCreate set to true: "elastic-agent-user"(S-1-5-21-74956108-3170057274-2946573734-1008): [0s]

[ ] Copying install files [0s]

[ ] Copying install files [0s]

[ ==] Copying install files [0s]

[ ===] Copying install files [0s]

[ ===] Copying install files [0s]

[=== ] Copying install files [0s]

[=== ] Copying install files [0s]

[== ] Copying install files [0s]

[ ] Copying install files [0s]

[ =] Copying install files [0s]

[ ==] Copying install files [1s]

[ ===] Copying install files [1s]

[====] Copying install files [1s]

[=== ] Copying install files [1s]

[== ] Copying install files [1s]

[= ] Copying install files [1s]

[ ] Copying install files [1s]

[ =] Copying install files [1s]

[ ==] Copying install files [1s]

[ ===] Copying install files [1s]

[====] Copying install files [2s]

[=== ] Copying install files [2s]

[== ] Copying install files [2s]

[= ] Copying install files [2s]

[= ] Successfully copied files [2s]

[= ] Successfully copied files [2s]

[ =] Successfully copied files [2s]

[ =] Installing service [2s]

[ ==] Installing service [2s]

[ ===] Installing service [2s]

[====] Installing service [2s]

[====] Installed service [2s]

[====] Starting Service [2s]

[====] Starting Service [2s]

[=== ] Service Started [2s] Elastic Agent successfully installed, starting enrollment.

[=== ] Enrolling Elastic Agent with Fleet [2s]

[=== ] Waiting For Enroll... [2s]

[== ] Waiting For Enroll... [3s] {"log.level":"info","@timestamp":"2026-08-26T22:50:41.131Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":87},"message":"Starting enrollment to URL: https://4c574a8d2ad35ef68edd7ca4cf765e2d.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}

[= ] Waiting For Enroll... [3s]

[ ] Waiting For Enroll... [3s]

[ =] Waiting For Enroll... [3s]

[ ==] Waiting For Enroll... [3s]

[ ===] Waiting For Enroll... [3s]

[ ===] Waiting For Enroll... [3s]

[====] Waiting For Enroll... [3s]

[=== ] Waiting For Enroll... [3s]

[== ] Waiting For Enroll... [3s]

[ ] Waiting For Enroll... [4s]

[ ] Waiting For Enroll... [4s]

[ ==] Waiting For Enroll... [4s]

[ ===] Waiting For Enroll... [4s]

[ ===] Waiting For Enroll... [4s]

[=== ] Waiting For Enroll... [4s] {"log.level":"info","@timestamp":"2026-08-26T22:50:42.640Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":417},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-08-26T22:50:42.644Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":206},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.

[=== ] Enroll Completed [4s]

[=== ] Done [4s]

[=== ] Done [4s]
Elastic Agent has been successfully installed.
upgrade_fleet_test.go:464: Waiting for Agent to be correct version and healthy...
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: agent state is not healthy: got 0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrade_fleet_test.go:474: Agent ID: "e66693fc-7041-4e4e-a3d1-ac43d60a8c09"
upgrade_fleet_test.go:476: Waiting for enrolled Agent status to be online...
upgrade_fleet_test.go:489: Upgrading from version "9.6.0-SNAPSHOT-540fc31f60c1cc58c5b448388ed70e69af69ac16" to version "9.6.0-SNAPSHOT-0fc03a8a0c522f168b0d8f8f79b35a75873b66f0"...
upgrade_fleet_test.go:497: Waiting from upgrade details to show up in Fleet
upgrade_fleet_test.go:513: Waiting for upgrade watcher to start...
upgrade_fleet_test.go:518: Upgrade watcher started
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
fixture.go:985: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
upgrader.go:573: waiting for healthy agent and proper version: commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
upgrade_fleet_test.go:364:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:364
C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:117
C:/buildkite-agent/builds/bk-agent-prod-gcp-1787780316924129446/elastic/elastic-agent/testing/integration/ess/upgrade_fleet_test.go:57
Error: Received unexpected error:
waiting for agent to be healthy and version 9.6.0-SNAPSHOT: failed waiting for healthy agent and version (context deadline exceeded): commits don't match: got 08cd0724bd5e734d13249f7986d330d67fb8ebf0, want 0fc03a8a0c522f168b0d8f8f79b35a75873b66f0
Test: TestFleetManagedUpgradeUnprivileged
fixture_install.go:325: [test TestFleetManagedUpgradeUnprivileged] Inside fixture cleanup function
fixture_install.go:341: collecting diagnostics; test failed
fixture_install.go:833: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1787780316924129446\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeUnprivileged-2026-08-26T22-53-37Z-diagnostics.zip]
fixture_install.go:784: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
fixture.go:1389: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1787780316924129446\elastic\elastic-agent\build\diagnostics\TestFleetManagedUpgradeUnprivileged-2026-08-26T22-53-37Z-ProcessDump-cleanup.json
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeUnprivileged360939604" preserved for investigation/debugging
fixture.go:1708: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestFleetManagedUpgradeUnprivileged215907771" preserved for investigation/debugging
--- FAIL: TestFleetManagedUpgradeUnprivileged (201.69s)

```

Contributor guide

Open the contributing guide

Research direction

Start with TestFleetManagedUpgradeUnprivileged in the github.com/elastic/elastic-agent/testing/integration/ess scope and review the failure output from the linked Buildkite run. Trace the upgrade flow through upgrade_fleet_test.go, fixture.go, fixture_install.go, and upgrader.go, focusing on the reported health and commit mismatches. Done means the test no longer flakes and reliably reaches the expected upgraded version.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.