elastic / elastic/elastic-agent
[Flaky Test]: TestPolicyChangePersistsWithoutAck – agent.policy_revision did not advance after policy update with acks disabled
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
### Failing test case
TestPolicyChangePersistsWithoutAck
### Error message
agent.policy_revision did not advance after policy update with acks disabled
### Build
https://buildkite.com/elastic/elastic-agent/builds/39259#019e2068-f5c8-4989-9ad4-87073b2dfaba
### OS
Windows
### Stacktrace and notes
```markdown
=== FAIL: testing/integration/ess TestPolicyChangePersistsWithoutAck (357.05s)
tools.go:121: Creating enrollment API key...
tools.go:112: Creating enrollment API key...
fixture_install.go:200: [test TestPolicyChangePersistsWithoutAck] Inside fixture install function
fixture_install.go:228: [test TestPolicyChangePersistsWithoutAck] Inside fixture installNoPkgManager function
fetcher.go:103: Using existing artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip
fixture.go:408: Extracting artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345
fixture.go:426: Completed extraction of artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345
fixture.go:1210: Components were not modified from the fetched artifact
fixture_install.go:247: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345\elastic-agent-9.3.5-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged --url https://f999f770291b1e78a50aff3e0b47c261.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token VVBlWElKNEJ6bmVYZnV3aTFLZWM6OXRwZEZlZWZFNVRVMS1tTzlHUGZUdw==]
tools.go:112: >>> Enroll succeeded. Output: Unprivileged installation mode enabled.
Installing in non-interactive mode.
[== ] Service Started [3s] Elastic Agent successfully installed, starting enrollment.
[== ] Waiting For Enroll... [3s] {"log.level":"info","@timestamp":"2026-05-13T09:08:12.161Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":86},"message":"Starting enrollment to URL: https://f999f770291b1e78a50aff3e0b47c261.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-05-13T09:08:13.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":395},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-05-13T09:08:13.656Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":205},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[= ] Done [4s]
Elastic Agent has been successfully installed.
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STOPPED
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STARTING
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STARTING
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:112: >>> Enrolled Agent ID: 6f2d35a7-bbe7-4c10-8682-5315c881513b
policy_change_no_ack_test.go:78: created policy 5de0e406-10f7-401f-aa30-84622fe3ff06, enrolled agent 6f2d35a7-bbe7-4c10-8682-5315c881513b
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
policy_change_no_ack_test.go:91: agent reported initial policy revision 1
policy_change_no_ack_test.go:108: waiting for agent to report policy revision 2
policy_change_no_ack_test.go:110:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1778660231940892760/elastic/elastic-agent/testing/integration/ess/policy_change_no_ack_test.go:115
C:/Users/Buildkite/.go/go-1.26.3/src/runtime/asm_amd64.s:1771
Error: "1" is not greater than or equal to "2"
Messages: agent.policy_revision did not advance after policy update with acks disabled; the POLICY_CHANGE action was either not persisted in the state store or its policy id/revision fields were not reported in the next checkin
policy_change_no_ack_test.go:110:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1778660231940892760/elastic/elastic-agent/testing/integration/ess/policy_change_no_ack_test.go:110
Error: Condition never satisfied
Test: TestPolicyChangePersistsWithoutAck
fixture_install.go:312: [test TestPolicyChangePersistsWithoutAck] Inside fixture cleanup function
fixture_install.go:328: collecting diagnostics; test failed
fixture_install.go:816: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1778660231940892760\elastic\elastic-agent\build\diagnostics\TestPolicyChangePersistsWithoutAck-2026-05-13T09-13-48Z-diagnostics.zip]
fixture_install.go:767: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
fixture.go:1341: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1778660231940892760\elastic\elastic-agent\build\diagnostics\TestPolicyChangePersistsWithoutAck-2026-05-13T09-13-48Z-ProcessDump-cleanup.json
fixture.go:1660: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestPolicyChangePersistsWithoutAck2209556345" preserved for investigation/debugging
```
Contributor guide
Research direction
Start with testing/integration/ess/policy_change_no_ack_test.go, especially TestPolicyChangePersistsWithoutAck and its wait for policy revision 2. Reproduce the Windows integration failure and inspect the diagnostic output and check-in behavior around the POLICY_CHANGE action. Done means the policy revision advances after the update with acknowledgements disabled and the test passes reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100