Removing maintenance mode causes consul to crash on windows server
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
We run consul agents as a windows service and are seeing intermittent issues where removing maintenance mode causes the service to crash.
#### Reproduction Steps
Issue is intermittent but reproducible
1. From the windows agent HTTP PUT http://localhost:8500/v1/agent/maintenance?enable=False&reason=I+am+a+reason
2. View error
### Consul info for both Client and Server
Client info
```
PS D:\consul> ./consul info
agent:
check_monitors = 5
check_ttls = 0
checks = 6
services = 5
build:
prerelease =
revision = 9ea1a204
version = 1.7.2
consul:
acl = enabled
known_servers = 5
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 56
max_procs = 4
os = windows
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 259
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 28098
members = 182
query_queue = 0
query_time = 1
```
Server info
```
consul info
agent:
check_monitors = 2
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 9ea1a204
version = 1.7.2
consul:
acl = enabled
bootstrap = false
known_datacenters = 4
leader = false
leader_addr = 172.25.8.44:8300
server = true
raft:
applied_index = 277427988
commit_index = 277427988
fsm_pending = 0
last_contact = 42.403813ms
last_log_index = 277427988
last_log_term = 2074
last_snapshot_index = 277424837
last_snapshot_term = 2074
latest_configuration = [{Suffrage:Voter ID:16d1708e-3a9e-bba1-db98-d0bcb51255e8 Address:172.25.8.44:8300} {Suffrage:Voter ID:97368e83-ae6d-0630-2f34-5482852156e4 Address:172.25.8.43:8300} {Suffrage:Voter ID:e828f4c9-562e-4d59-9bef-0eef0c924f4e Address:172.25.8.41:8300} {Suffrage:Voter ID:a1b71015-4b56-6923-13b7-94969f815f71 Address:172.25.8.40:8300} {Suffrage:Voter ID:c6231866-1421-8868-28ce-eb995fa49e02 Address:172.25.8.42:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2074
runtime:
arch = amd64
cpu_count = 2
goroutines = 913
max_procs = 2
os = linux
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 323
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 33560
members = 342
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1212
members = 20
query_queue = 0
query_time = 1
```
### Operating system and Environment details
Windows Server 2012 agent
RHEL 7.7 server
### Log Fragments
I couldn't find any relevant logs on the server side, but on the agent side the following is logged.
Exception 0xc0000005 0x0 0xc00049e000 0x7fff50392f6a
PC=0x7fff50392f6a
syscall.Syscall(0x7fff503928b0, 0x2, 0xc00049e000, 0xc000144050, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/runtime/syscall_windows.go:188 +0xfa
syscall.(*Proc).Call(0xc000275480, 0xc0001422c0, 0x2, 0x2, 0x1e8, 0x20, 0x8, 0xc000177b00)
/usr/local/go/src/syscall/dll_windows.go:173 +0x1f0
github.com/go-ole/go-ole.CLSIDFromProgID(0x310898c, 0x1a, 0x19dc6b6, 0xc000275400, 0xc000144040)
/go/pkg/mod/github.com/go-ole/go-ole@v1.2.1/com.go:121 +0xb1
github.com/go-ole/go-ole.ClassIDFrom(0x310898c, 0x1a, 0x431e41, 0x5176ac0, 0x0)
/go/pkg/mod/github.com/go-ole/go-ole@v1.2.1/utility.go:14 +0x40
github.com/go-ole/go-ole/oleutil.CreateObject(0x310898c, 0x1a, 0x0, 0x1, 0x37065c0)
/go/pkg/mod/github.com/go-ole/go-ole@v1.2.1/oleutil/oleutil.go:16 +0x40
github.com/StackExchange/wmi.(*Client).Query(0x514bce0, 0xc000370240, 0x11d, 0x276f8a0, 0xc000275ac0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/!stack!exchange/wmi@v0.0.0-20180116203802-5d049714c4a6/wmi.go:150 +0x2d3
github.com/StackExchange/wmi.Query(0xc000370240, 0x11d, 0x276f8a0, 0xc000275ac0, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/!stack!exchange/wmi@v0.0.0-20180116203802-5d049714c4a6/wmi.go:76 +0x10c
github.com/shirou/gopsutil/internal/common.WMIQueryWithContext.func1(0xc00035a540, 0xc000370240, 0x11d, 0x276f8a0, 0xc000275ac0, 0x0, 0x0, 0x0)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/internal/common/common_windows.go:126 +0x81
created by github.com/shirou/gopsutil/internal/common.WMIQueryWithContext
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/internal/common/common_windows.go:125 +0x122
goroutine 1 [select, locked to thread]:
github.com/shirou/gopsutil/internal/common.WMIQueryWithContext(0x36b4840, 0xc00035a4e0, 0xc000370240, 0x11d, 0x276f8a0, 0xc000275ac0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/internal/common/common_windows.go:129 +0x1d4
github.com/shirou/gopsutil/cpu.perfInfoWithContext(0x36b4800, 0xc000060130, 0xc000453bb0, 0x44fa62, 0x463230, 0x5154fb0, 0xc000000000)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/cpu/cpu_windows.go:126 +0xba
github.com/shirou/gopsutil/cpu.perCPUTimesWithContext(0x36b4800, 0xc000060130, 0x0, 0x0, 0x1, 0xc0003c9810, 0x0)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/cpu/cpu_windows.go:153 +0x57
github.com/shirou/gopsutil/cpu.TimesWithContext(0x36b4800, 0xc000060130, 0x306e701, 0xc0003c9810, 0x1, 0x1, 0x0, 0x0)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/cpu/cpu_windows.go:54 +0x311
github.com/shirou/gopsutil/cpu.Times(...)
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/cpu/cpu_windows.go:49
github.com/shirou/gopsutil/cpu.init.0()
/go/pkg/mod/github.com/shirou/gopsutil@v0.0.0-20181107111621-48177ef5f880/cpu/cpu.go:62 +0xb8
goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0xa3
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x48
goroutine 9 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x5153d80)
/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x92
created by github.com/golang/glog.init.0
/go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x276
rax 0x1
rbx 0x7fff50392f40
rcx 0x8c2fcc8
rdi 0x7fff504b74b0
rsi 0x7fff504b2620
rbp 0xc00049e000
rsp 0x8c2fc60
r8 0x0
r9 0x8c2fcc8
r10 0xc00049e000
r11 0x0
r12 0x0
r13 0x7fff504b74b0
r14 0x8c2fd68
r15 0x7fff503d7af8
rip 0x7fff50392f6a
rflags 0x10202
cs 0x33
fs 0x53
gs 0x2b
Contributor guide
Assessment
This issue has not been assessed yet.