linkedin / linkedin/Burrow

notifier: group state is mutated under read locks from concurrent goroutines (potential fatal concurrent map write)

Open
#869 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
818
Avg merge
1h 14m
Merged PRs (30d)
1

Description

### Summary

`checkAndSendResponseToModules` is spawned as a goroutine per evaluator response (`responseLoop`, coordinator.go:407), so multiple instances run concurrently by design. Each holds only **read** locks (`nc.clusterLock.RLock()` + `cluster.Lock.RLock()`) while mutating shared per-group state:

- `cgroup.ID` and `cgroup.Start` are written at coordinator.go:430-434 and 454-458 under the read lock.
- `notifyModule` (called synchronously from there, still under the same read locks) **writes to the `cgroup.LastNotify` map** at coordinator.go:548 and 566.

Read locks don't exclude each other, so two concurrent responses for the same group can write `cgroup.ID`/`Start` racily and — worse — perform concurrent writes to the same `LastNotify` map. Concurrent map writes are a **fatal, unrecoverable runtime error** in Go (`fatal error: concurrent map writes`), which would take down the whole Burrow process.

Two responses for the same group can be in flight at once, e.g. when two `sendEvaluatorRequests` loops briefly overlap after a ZooKeeper session bounce (`manageEvalLoop` starts a new loop once it reacquires the ZK lock while the old one is still mid-iteration), or whenever response processing is slower than the evaluation interval.

### Why not just a PR

The minimal fix — upgrading `cluster.Lock.RLock()` to `Lock()` in `checkAndSendResponseToModules` — is a one-line change, but it would hold the cluster's write lock across `module.Notify()` calls (HTTP/email sends), serializing notification dispatch per cluster and blocking the evaluation loop for that cluster while a slow notifier runs. Alternatives:

1. Accept the serialization (correctness over throughput; notify calls are usually fast).
2. Restructure so the `LastNotify`/`ID`/`Start` check-and-set happens under a short write lock and `Notify()` is called after release (the status/startTime/eventID are already passed by value).
3. Give `consumerGroup` its own small mutex for `ID`/`Start`/`LastNotify`.

Option 2 seems cleanest but touches the send-once/send-interval logic, so I wanted maintainer input on the preferred direction before sending a PR. Happy to implement whichever is preferred.

Related: #867 fixes the same pattern (write under read lock) in `sendEvaluatorRequests`, where the fix has no locking tradeoff.

Burrow version: master (60d5782)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in coordinator.go at responseLoop (around line 407), checkAndSendResponseToModules, and notifyModule, then compare the related locking fix in issue #867. Determine how the ID, Start, and LastNotify check-and-set can be made safe without leaving notification dispatch under an unnecessarily broad lock; done means concurrent responses cannot race or trigger fatal map writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.