elastic / elastic/elastic-agent
Windows 11: Upgrade TestCleanup fails with Unlock of unlocked RWMutex
- Dominant language
- Go
- Stars
- 276
- Forks
- 266
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 312
Description
https://buildkite.com/elastic/elastic-agent/builds/37861#019dbb7d-afb1-4c99-9cf7-29cf7464e6ec
```
=== Failed
=== FAIL: internal/pkg/agent/application/upgrade TestCleanup (unknown)
fatal error: sync: Unlock of unlocked RWMutex
goroutine 125 [running]:
sync.fatal({0x1414c3969?, 0xc040000000?})
C:/Users/Buildkite/.go/go-1.25.9/src/runtime/panic.go:1026 +0x18
sync.(*RWMutex).Unlock(0x23c000)
C:/Users/Buildkite/.go/go-1.25.9/src/sync/rwmutex.go:212 +0x7e
testing.tRunner(0x23c000, 0xc0001ea090)
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:1939 +0x1f2
runtime: g 125: unexpected return pc for testing.tRunner called from 0x7ffed076a590
stack: frame={sp:0xc0001e9ef0, fp:0xc0001e9fb0} stack=[0xc0001e6000,0xc0001ea000)
0x000000c0001e9df0: 0x000000c0001e9cc0 0x000000c00047a460
0x000000c0001e9e00: 0x000000014008da9b 0x000000014008a337
0x000000c0001e9e10: 0x000000c0001e9e50 0x0000000140049e78
0x000000c0001e9e20: 0x000000c0001e9e30 0x000000c000003c00
0x000000c0001e9e30: 0x0000000140049ea0 0x000000c000003c00
0x000000c0001e9e40: 0x0000000140049a85 0x000000c0001e9e60
0x000000c0001e9e50: 0x000000c0001e9e80 0x0000000140049a85
0x000000c0001e9e60: 0x000000c0001e9e68 0x0000000140049ac0
0x000000c0001e9e70: 0x00000001414c3969 0x0000000000000020
0x000000c0001e9e80: 0x000000c0001e9ea0 0x0000000140083798
0x000000c0001e9e90: 0x00000001414c3969 0x0000000000000020
0x000000c0001e9ea0: 0x000000c0001e9ee0 0x00000001400b451e
0x000000c0001e9eb0: 0x00000001414c3969 0x000000c040000000
0x000000c0001e9ec0: 0x00000001400b438f 0x400000000023c014
0x000000c0001e9ed0: 0x0000000140000000 0x000000000023c00c
0x000000c0001e9ee0: 0x000000c0001e9fa0 0x00000001402b7152
0x000000c0001e9ef0: <0x000000000023c000 0x00000006b9d722aa
0x000000c0001e9f00: 0x000000000000000f 0x0000000000000000
0x000000c0001e9f10: 0x000000c0001e9ef0 0x00000001402b70b4
0x000000c0001e9f20: 0x000000014008a337 0x000000c0001e9f88
0x000000c0001e9f30: 0x00000001400633d1 0x000000c0001e9f60
0x000000c0001e9f40: 0x000000c0001e9ef0 0x00000001402b705b
0x000000c0001e9f50: 0xffffffff001e9f68 0x000000c000002000
0x000000c0001e9f60: 0x00000001400634a0 0x000000c0001e9f54
0x000000c0001e9f70: 0x00000001414fe480 0x000000c0001ea0e0
0x000000c0001e9f80: 0x000000c000002000 0x000000c0001ea090
0x000000c0001e9f90: 0x000000014008e34a 0x000000c0001ea0e0
0x000000c0001e9fa0: 0x000000c0001e9fd0 0x00007ffed076a590
0x000000c0001e9fb0: >0x000000000023c000 0x000000c0001ea090
0x000000c0001e9fc0: 0x000000c0001ea800 0x000000c0001ea380
0x000000c0001e9fd0: 0x000000c0001ea870 0x00007ffed076a598
0x000000c0001e9fe0: 0x00000000007b46b0 0x0000000000000006
0x000000c0001e9ff0: 0x0000000000000000 0x0000000000000000
created by testing.(*T).Run in goroutine 1
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:1997 +0x9dd
goroutine 1 [chan receive]:
testing.(*T).Run(0xc000003880, {0x1414a7bd7, 0xb}, 0x1414fdbf8)
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:2005 +0xa08
testing.runTests.func1(0xc000003880)
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:2477 +0x86
testing.tRunner(0xc000003880, 0xc00035fae0)
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:1934 +0x1c4
testing.runTests(0xc0000089c0, {0x14121b180, 0x2f, 0x2f}, {0x0?, 0x140080fe5?, 0x141ebdbc0?})
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:2475 +0x96d
testing.(*M).Run(0xc0002b7ae0)
C:/Users/Buildkite/.go/go-1.25.9/src/testing/testing.go:2337 +0xed5
main.main()
_testmain.go:147 +0x165
goroutine 132 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 131
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 120 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 119
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 117 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 116
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 77 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 76
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 81 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 80
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 100 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 99
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 74 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 73
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
goroutine 103 [select]:
github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1.1()
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:248 +0x14a
created by github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.TestProcessMarker.func1 in goroutine 102
C:/buildkite-agent/builds/bk-agent-prod-azure-1776967305487486889/elastic/elastic-agent/internal/pkg/agent/application/upgrade/marker_watcher_test.go:246 +0x47d
rollback_test.go:664: Created fake agent install for {version:{version:0.9.9 hash:aaaaaa} useVersionInPath:false} located at "data\\elastic-agent-aaaaaa"
rollback_test.go:664: Created fake agent install for {version:{version:1.1.1 hash:aaabbb} useVersionInPath:false} located at "data\\elastic-agent-aaabbb"
rollback_test.go:664: Created fake agent install for {version:{version:1.2.3-SNAPSHOT hash:abcdef} useVersionInPath:false} located at "data\\elastic-agent-abcdef"
rollback_test.go:666: Setting version {version:1.2.3-SNAPSHOT hash:abcdef} as FROM version for the update marker
rollback_test.go:664: Created fake agent install for {version:{version:4.5.6-SNAPSHOT hash:ghijkl} useVersionInPath:true} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:672: Setting version {version:4.5.6-SNAPSHOT hash:ghijkl} as TO version for the update marker
rollback_test.go:679: Creating symlink pointing to "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:685: Creating upgrade marker from {version:1.2.3-SNAPSHOT hash:abcdef} located at "data\\elastic-agent-abcdef" to {version:4.5.6-SNAPSHOT hash:ghijkl} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:221: Loaded update marker &{Version:4.5.6-SNAPSHOT Hash:ghijkl VersionedHome:data\elastic-agent-4.5.6-SNAPSHOT-ghijkl UpdatedOn:2026-04-23 18:21:13.5034434 +0000 UTC PrevVersion:1.2.3-SNAPSHOT PrevHash:abcdef PrevVersionedHome:data\elastic-agent-abcdef Acked:false Action: Details: RollbacksAvailable:map[]}
PASS: TestCleanup/cleanup_with_versionedHome_only_on_the_new_agent_+_extra_old_agent_installs (0.03s)
rollback_test.go:664: Created fake agent install for {version:{version:1.2.3-SNAPSHOT hash:abcdef} useVersionInPath:false} located at "data\\elastic-agent-abcdef"
rollback_test.go:666: Setting version {version:1.2.3-SNAPSHOT hash:abcdef} as FROM version for the update marker
rollback_test.go:664: Created fake agent install for {version:{version:4.5.6-SNAPSHOT hash:ghijkl} useVersionInPath:false} located at "data\\elastic-agent-ghijkl"
rollback_test.go:672: Setting version {version:4.5.6-SNAPSHOT hash:ghijkl} as TO version for the update marker
rollback_test.go:679: Creating symlink pointing to "data\\elastic-agent-ghijkl"
rollback_test.go:685: Creating upgrade marker from {version:1.2.3-SNAPSHOT hash:abcdef} located at "data\\elastic-agent-abcdef" to {version:4.5.6-SNAPSHOT hash:ghijkl} located at "data\\elastic-agent-ghijkl"
rollback_test.go:221: Loaded update marker &{Version: Hash:ghijkl VersionedHome: UpdatedOn:2026-04-23 18:21:13.5197002 +0000 UTC PrevVersion:1.2.3-SNAPSHOT PrevHash:abcdef PrevVersionedHome: Acked:false Action: Details: RollbacksAvailable:map[]}
PASS: TestCleanup/cleanup_without_versionedHome_(legacy_upgrade_process) (0.01s)
rollback_test.go:664: Created fake agent install for {version:{version:1.2.3-SNAPSHOT hash:abcdef} useVersionInPath:true} located at "data\\elastic-agent-1.2.3-SNAPSHOT-abcdef"
rollback_test.go:666: Setting version {version:1.2.3-SNAPSHOT hash:abcdef} as FROM version for the update marker
rollback_test.go:664: Created fake agent install for {version:{version:4.5.6-SNAPSHOT hash:ghijkl} useVersionInPath:true} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:672: Setting version {version:4.5.6-SNAPSHOT hash:ghijkl} as TO version for the update marker
rollback_test.go:679: Creating symlink pointing to "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:685: Creating upgrade marker from {version:1.2.3-SNAPSHOT hash:abcdef} located at "data\\elastic-agent-1.2.3-SNAPSHOT-abcdef" to {version:4.5.6-SNAPSHOT hash:ghijkl} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:221: Loaded update marker &{Version:4.5.6-SNAPSHOT Hash:ghijkl VersionedHome:data\elastic-agent-4.5.6-SNAPSHOT-ghijkl UpdatedOn:2026-04-23 18:21:13.5320143 +0000 UTC PrevVersion:1.2.3-SNAPSHOT PrevHash:abcdef PrevVersionedHome:data\elastic-agent-1.2.3-SNAPSHOT-abcdef Acked:false Action: Details: RollbacksAvailable:map[]}
PASS: TestCleanup/cleanup_with_versionedHome_(new_upgrade_process) (0.01s)
rollback_test.go:664: Created fake agent install for {version:{version:1.2.3-SNAPSHOT hash:abcdef} useVersionInPath:false} located at "data\\elastic-agent-abcdef"
rollback_test.go:666: Setting version {version:1.2.3-SNAPSHOT hash:abcdef} as FROM version for the update marker
rollback_test.go:664: Created fake agent install for {version:{version:4.5.6-SNAPSHOT hash:ghijkl} useVersionInPath:true} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:672: Setting version {version:4.5.6-SNAPSHOT hash:ghijkl} as TO version for the update marker
rollback_test.go:679: Creating symlink pointing to "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:685: Creating upgrade marker from {version:1.2.3-SNAPSHOT hash:abcdef} located at "data\\elastic-agent-abcdef" to {version:4.5.6-SNAPSHOT hash:ghijkl} located at "data\\elastic-agent-4.5.6-SNAPSHOT-ghijkl"
rollback_test.go:221: Loaded update marker &{Version:4.5.6-SNAPSHOT Hash:ghijkl VersionedHome:data\elastic-agent-4.5.6-SNAPSHOT-ghijkl UpdatedOn:2026-04-23 18:21:13.5510638 +0000 UTC PrevVersion:1.2.3-SNAPSHOT PrevHash:abcdef PrevVersionedHome:data\elastic-agent-abcdef Acked:false Action: Details: RollbacksAvailable:map[]}
PASS: TestCleanup/cleanup_with_versionedHome_only_on_the_new_agent_(new_upgrade_process_from_an_old_agent_upgraded_with_legacy_process) (0.02s)
```
Contributor guide
Research direction
Start with internal/pkg/agent/application/upgrade/rollback_test.go and the TestCleanup cases shown in the failure, then inspect the marker watcher goroutines in marker_watcher_test.go. Reproduce the failure on Windows 11 and trace the RWMutex ownership around cleanup and watcher shutdown. Done means TestCleanup completes without the unlocked-RWMutex panic and the upgrade tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100