elastic / elastic/beats

[otel-beatreceiver-isolation] Concurrent Filebeat receiver construction can panic on global nomad metadata map writes

Open
#50,029 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 1h
Merged PRs (30d)
370

Description

## Findings

### 1. Concurrent construction of multiple Filebeat receivers can panic (`concurrent map writes`)

**Location**
- `x-pack/filebeat/fbreceiver/factory.go:41-43`
- `x-pack/libbeat/cmd/instance/beat.go:56-60`
- `libbeat/cmd/instance/beat.go:214-218`
- `x-pack/filebeat/include/list.go:64-67`
- `x-pack/filebeat/processors/add_nomad_metadata/matchers.go:35-39`
- `x-pack/libbeat/processors/add_nomad_metadata/indexing.go:53-59`

**Evidence**
Receiver construction appends and executes module initialization callbacks on every construction:

- `settings.Initialize = append(settings.Initialize, include.InitializeModule)` (`x-pack/filebeat/fbreceiver/factory.go:41`)
- `instance.NewBeat(..., settings.Initialize)` (`x-pack/libbeat/cmd/instance/beat.go:56-60`)
- `for _, f := range initFuncs { f() }` (`libbeat/cmd/instance/beat.go:216-218`)

`include.InitializeModule()` calls the Nomad metadata initializer every time:

- `m0.InitializeModule()` (`x-pack/filebeat/include/list.go:65`)

That initializer writes default matcher/indexer configs into global maps:

- `add_nomad_metadata.Indexing.AddDefaultIndexerConfig(...)` (`x-pack/filebeat/processors/add_nomad_metadata/matchers.go:35-36`)
- `add_nomad_metadata.Indexing.AddDefaultMatcherConfig(...)` (`x-pack/filebeat/processors/add_nomad_metadata/matchers.go:39`)

Those writes are unsynchronized in the global register:

- `r.defaultIndexerConfigs[name] = config` (`x-pack/libbeat/processors/add_nomad_metadata/indexing.go:54`)
- `r.defaultMatcherConfigs[name] = config` (`x-pack/libbeat/processors/add_nomad_metadata/indexing.go:59`)

Unlike `AddMatcher`/`AddIndexer`, `AddDefault*` does not lock.

**What is wrong**
`add_nomad_metadata.Indexing` is a package-global singleton, and receiver construction executes this initializer for each receiver instance. If two `filebeatreceiver` instances are created concurrently in the same OTel Collector process, unsynchronized writes to the same maps can race and panic.

**Why it matters (high severity)**
A real Collector deployment with multiple Beat receivers (or concurrent restart/recreate flows) can crash the Collector process during receiver construction due to `fatal error: concurrent map writes`.

**Suggested fix direction**
1. Make the `add_nomad_metadata` module initialization idempotent and one-time (e.g., `sync.Once` around default registration), and/or
2. Add locking to `AddDefaultIndexerConfig` and `AddDefaultMatcherConfig` to match the locking behavior used by `AddIndexer`/`AddMatcher`.

A robust fix should do both: guard one-time registration and ensure map writes are synchronized.

## Already tracked / safe summary

The previously tracked constructor-global issues (`config` resolver overwrite, metricbeat path singleton, timestamp precision singleton, `underAgent`/feature globals) remain covered by existing `otel-beatreceiver-isolation` issues. This report is only for the distinct concurrent map-write panic path above.

> [!NOTE]
>
> 🔒 Integrity filtering filtered 4 items
>
> Integrity filtering activated and filtered the following items during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - [#49658](https://github.com/elastic/beats/issues/49658) (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:elastic/beats#49610 (`issue_read`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:elastic/beats#49877 (`issue_read`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
> - issue:#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: OTel BeatReceiver Global State Isolation](https://github.com/elastic/beats/actions/runs/24183336817)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 16, 2026, 10:01 AM UTC

Contributor guide

Open the contributing guide

Research direction

Start with x-pack/filebeat/fbreceiver/factory.go and trace initialization through x-pack/filebeat/include/list.go into x-pack/libbeat/processors/add_nomad_metadata/indexing.go. Compare AddDefaultIndexerConfig and AddDefaultMatcherConfig with the locked registration methods, then inspect existing receiver-construction tests. Done means concurrent Filebeat receiver construction no longer races or panics and the behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.