notifier: group state is mutated under read locks from concurrent goroutines (potential fatal concurrent map write)
- Dominant language
- Go
- Stars
- 4k
- Forks
- 818
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Description
### Summary
`checkAndSendResponseToModules` is spawned as a goroutine per evaluator response (`responseLoop`, coordinator.go:407), so multiple instances run concurrently by design. Each holds only **read** locks (`nc.clusterLock.RLock()` + `cluster.Lock.RLock()`) while mutating shared per-group state:
- `cgroup.ID` and `cgroup.Start` are written at coordinator.go:430-434 and 454-458 under the read lock.
- `notifyModule` (called synchronously from there, still under the same read locks) **writes to the `cgroup.LastNotify` map** at coordinator.go:548 and 566.
Read locks don't exclude each other, so two concurrent responses for the same group can write `cgroup.ID`/`Start` racily and — worse — perform concurrent writes to the same `LastNotify` map. Concurrent map writes are a **fatal, unrecoverable runtime error** in Go (`fatal error: concurrent map writes`), which would take down the whole Burrow process.
Two responses for the same group can be in flight at once, e.g. when two `sendEvaluatorRequests` loops briefly overlap after a ZooKeeper session bounce (`manageEvalLoop` starts a new loop once it reacquires the ZK lock while the old one is still mid-iteration), or whenever response processing is slower than the evaluation interval.
### Why not just a PR
The minimal fix — upgrading `cluster.Lock.RLock()` to `Lock()` in `checkAndSendResponseToModules` — is a one-line change, but it would hold the cluster's write lock across `module.Notify()` calls (HTTP/email sends), serializing notification dispatch per cluster and blocking the evaluation loop for that cluster while a slow notifier runs. Alternatives:
1. Accept the serialization (correctness over throughput; notify calls are usually fast).
2. Restructure so the `LastNotify`/`ID`/`Start` check-and-set happens under a short write lock and `Notify()` is called after release (the status/startTime/eventID are already passed by value).
3. Give `consumerGroup` its own small mutex for `ID`/`Start`/`LastNotify`.
Option 2 seems cleanest but touches the send-once/send-interval logic, so I wanted maintainer input on the preferred direction before sending a PR. Happy to implement whichever is preferred.
Related: #867 fixes the same pattern (write under read lock) in `sendEvaluatorRequests`, where the fix has no locking tradeoff.
Burrow version: master (60d5782)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in coordinator.go at responseLoop (around line 407), checkAndSendResponseToModules, and notifyModule, then compare the related locking fix in issue #867. Determine how the ID, Start, and LastNotify check-and-set can be made safe without leaving notification dispatch under an unnecessarily broad lock; done means concurrent responses cannot race or trigger fatal map writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100