[otel-beatreceiver-isolation] Concurrent Filebeat receiver construction can panic on global nomad metadata map writes
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 370
Description
## Findings
### 1. Concurrent construction of multiple Filebeat receivers can panic (`concurrent map writes`)
**Location**
- `x-pack/filebeat/fbreceiver/factory.go:41-43`
- `x-pack/libbeat/cmd/instance/beat.go:56-60`
- `libbeat/cmd/instance/beat.go:214-218`
- `x-pack/filebeat/include/list.go:64-67`
- `x-pack/filebeat/processors/add_nomad_metadata/matchers.go:35-39`
- `x-pack/libbeat/processors/add_nomad_metadata/indexing.go:53-59`
**Evidence**
Receiver construction appends and executes module initialization callbacks on every construction:
- `settings.Initialize = append(settings.Initialize, include.InitializeModule)` (`x-pack/filebeat/fbreceiver/factory.go:41`)
- `instance.NewBeat(..., settings.Initialize)` (`x-pack/libbeat/cmd/instance/beat.go:56-60`)
- `for _, f := range initFuncs { f() }` (`libbeat/cmd/instance/beat.go:216-218`)
`include.InitializeModule()` calls the Nomad metadata initializer every time:
- `m0.InitializeModule()` (`x-pack/filebeat/include/list.go:65`)
That initializer writes default matcher/indexer configs into global maps:
- `add_nomad_metadata.Indexing.AddDefaultIndexerConfig(...)` (`x-pack/filebeat/processors/add_nomad_metadata/matchers.go:35-36`)
- `add_nomad_metadata.Indexing.AddDefaultMatcherConfig(...)` (`x-pack/filebeat/processors/add_nomad_metadata/matchers.go:39`)
Those writes are unsynchronized in the global register:
- `r.defaultIndexerConfigs[name] = config` (`x-pack/libbeat/processors/add_nomad_metadata/indexing.go:54`)
- `r.defaultMatcherConfigs[name] = config` (`x-pack/libbeat/processors/add_nomad_metadata/indexing.go:59`)
Unlike `AddMatcher`/`AddIndexer`, `AddDefault*` does not lock.
**What is wrong**
`add_nomad_metadata.Indexing` is a package-global singleton, and receiver construction executes this initializer for each receiver instance. If two `filebeatreceiver` instances are created concurrently in the same OTel Collector process, unsynchronized writes to the same maps can race and panic.
**Why it matters (high severity)**
A real Collector deployment with multiple Beat receivers (or concurrent restart/recreate flows) can crash the Collector process during receiver construction due to `fatal error: concurrent map writes`.
**Suggested fix direction**
1. Make the `add_nomad_metadata` module initialization idempotent and one-time (e.g., `sync.Once` around default registration), and/or
2. Add locking to `AddDefaultIndexerConfig` and `AddDefaultMatcherConfig` to match the locking behavior used by `AddIndexer`/`AddMatcher`.
A robust fix should do both: guard one-time registration and ensure map writes are synchronized.
## Already tracked / safe summary
The previously tracked constructor-global issues (`config` resolver overwrite, metricbeat path singleton, timestamp precision singleton, `underAgent`/feature globals) remain covered by existing `otel-beatreceiver-isolation` issues. This report is only for the distinct concurrent map-write panic path above.
> [!NOTE]
>
> 🔒 Integrity filtering filtered 4 items
>
> Integrity filtering activated and filtered the following items during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - [#49658](https://github.com/elastic/beats/issues/49658) (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:elastic/beats#49610 (`issue_read`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:elastic/beats#49877 (`issue_read`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: OTel BeatReceiver Global State Isolation](https://github.com/elastic/beats/actions/runs/24183336817)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 16, 2026, 10:01 AM UTC
Contributor guide
Research direction
Start with x-pack/filebeat/fbreceiver/factory.go and trace initialization through x-pack/filebeat/include/list.go into x-pack/libbeat/processors/add_nomad_metadata/indexing.go. Compare AddDefaultIndexerConfig and AddDefaultMatcherConfig with the locked registration methods, then inspect existing receiver-construction tests. Done means concurrent Filebeat receiver construction no longer races or panics and the behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100