elastic / elastic/elastic-agent

[Flaky Test]: TestPolicyChangePersistsWithoutAck – agent.policy_revision did not advance after policy update with acks disabled

Open
#14,249 3 comments 0 reactions 0 assignees View on GitHub
flaky-test Team:Elastic-Agent
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

### Failing test case

TestPolicyChangePersistsWithoutAck

### Error message

agent.policy_revision did not advance after policy update with acks disabled

### Build

https://buildkite.com/elastic/elastic-agent/builds/39259#019e2068-f5c8-4989-9ad4-87073b2dfaba

### OS

Windows

### Stacktrace and notes

```markdown
=== FAIL: testing/integration/ess TestPolicyChangePersistsWithoutAck (357.05s)
tools.go:121: Creating enrollment API key...
tools.go:112: Creating enrollment API key...
fixture_install.go:200: [test TestPolicyChangePersistsWithoutAck] Inside fixture install function
fixture_install.go:228: [test TestPolicyChangePersistsWithoutAck] Inside fixture installNoPkgManager function
fetcher.go:103: Using existing artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip
fixture.go:408: Extracting artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345
fixture.go:426: Completed extraction of artifact elastic-agent-9.3.5-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345
fixture.go:1210: Components were not modified from the fetched artifact
fixture_install.go:247: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestPolicyChangePersistsWithoutAck2209556345\elastic-agent-9.3.5-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force --non-interactive --unprivileged --url https://f999f770291b1e78a50aff3e0b47c261.fleet.us-west2.gcp.elastic-cloud.com:443 --enrollment-token VVBlWElKNEJ6bmVYZnV3aTFLZWM6OXRwZEZlZWZFNVRVMS1tTzlHUGZUdw==]
tools.go:112: >>> Enroll succeeded. Output: Unprivileged installation mode enabled.
Installing in non-interactive mode.
[== ] Service Started [3s] Elastic Agent successfully installed, starting enrollment.
[== ] Waiting For Enroll... [3s] {"log.level":"info","@timestamp":"2026-05-13T09:08:12.161Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/enroll.EnrollWithBackoff","file.name":"enroll/enroll.go","file.line":86},"message":"Starting enrollment to URL: https://f999f770291b1e78a50aff3e0b47c261.fleet.us-west2.gcp.elastic-cloud.com:443/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-05-13T09:08:13.653Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).daemonReloadWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":395},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2026-05-13T09:08:13.656Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).Execute","file.name":"cmd/enroll_cmd.go","file.line":205},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[= ] Done [4s]
Elastic Agent has been successfully installed.
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STOPPED
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STARTING
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:187: wanted fleet status to be HEALTHY, was STARTING
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
tools.go:112: >>> Enrolled Agent ID: 6f2d35a7-bbe7-4c10-8682-5315c881513b
policy_change_no_ack_test.go:78: created policy 5de0e406-10f7-401f-aa30-84622fe3ff06, enrolled agent 6f2d35a7-bbe7-4c10-8682-5315c881513b
fixture.go:955: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
policy_change_no_ack_test.go:91: agent reported initial policy revision 1
policy_change_no_ack_test.go:108: waiting for agent to report policy revision 2
policy_change_no_ack_test.go:110:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1778660231940892760/elastic/elastic-agent/testing/integration/ess/policy_change_no_ack_test.go:115
C:/Users/Buildkite/.go/go-1.26.3/src/runtime/asm_amd64.s:1771
Error: "1" is not greater than or equal to "2"
Messages: agent.policy_revision did not advance after policy update with acks disabled; the POLICY_CHANGE action was either not persisted in the state store or its policy id/revision fields were not reported in the next checkin
policy_change_no_ack_test.go:110:
Error Trace: C:/buildkite-agent/builds/bk-agent-prod-gcp-1778660231940892760/elastic/elastic-agent/testing/integration/ess/policy_change_no_ack_test.go:110
Error: Condition never satisfied
Test: TestPolicyChangePersistsWithoutAck
fixture_install.go:312: [test TestPolicyChangePersistsWithoutAck] Inside fixture cleanup function
fixture_install.go:328: collecting diagnostics; test failed
fixture_install.go:816: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1778660231940892760\elastic\elastic-agent\build\diagnostics\TestPolicyChangePersistsWithoutAck-2026-05-13T09-13-48Z-diagnostics.zip]
fixture_install.go:767: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
fixture.go:1341: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1778660231940892760\elastic\elastic-agent\build\diagnostics\TestPolicyChangePersistsWithoutAck-2026-05-13T09-13-48Z-ProcessDump-cleanup.json
fixture.go:1660: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestPolicyChangePersistsWithoutAck2209556345" preserved for investigation/debugging
```

Contributor guide

Open the contributing guide

Research direction

Start with testing/integration/ess/policy_change_no_ack_test.go, especially TestPolicyChangePersistsWithoutAck and its wait for policy revision 2. Reproduce the Windows integration failure and inspect the diagnostic output and check-in behavior around the POLICY_CHANGE action. Done means the policy revision advances after the update with acknowledgements disabled and the test passes reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.