elastic / elastic/beats

[otel-beatreceiver-isolation] Concurrent filebeatreceiver construction races on global processor namespace registration

Open
#51,101 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 54m
Merged PRs (30d)
381

Description

## Findings

### 1. Filebeat receiver init re-registers processor plugins into unsynchronized global maps

**Location**
- `x-pack/filebeat/fbreceiver/factory.go:41`
- `x-pack/libbeat/cmd/instance/beat.go:56-60`
- `libbeat/cmd/instance/beat.go:214-218`
- `x-pack/filebeat/include/list.go:64-67`
- `x-pack/filebeat/processors/aws_vpcflow/parse_aws_vpc_flow_log.go:28-31`
- `x-pack/filebeat/processors/decode_cef/decode_cef.go:27-29`
- `libbeat/processors/registry.go:55-60`
- `libbeat/processors/script/javascript/module/processor/registry/registry.go:27,30-36`
- `libbeat/processors/namespace.go:30-33,53-59,67-72,75-92`

**Evidence**
- Receiver construction appends module initializer for every receiver instance:
- `settings.Initialize = append(settings.Initialize, include.InitializeModule)`
- Receiver construction executes all initializers each time:
- `instance.NewBeat(..., settings.Initialize)`
- `for _, f := range initFuncs { f() }`
- Filebeat module initializer always calls processor registration:
- `m1.InitializeModule()` and `m2.InitializeModule()` from `include.InitializeModule`
- `processors.RegisterPlugin(procName, New)` / `jsprocessor.RegisterPlugin(...)`
- Processor registries are package globals backed by a plain map with no lock:
- `var registry = NewNamespace()`
- `type Namespace struct { reg map[string]pluginer }`
- writes: `ns.reg[name] = p` (both leaf and namespace-path branches)

**What is wrong**
`filebeatreceiver` construction performs global plugin registration at runtime. When two receivers are constructed concurrently in the same Collector process, both goroutines can mutate/read the same `Namespace.reg` map without synchronization.

**Why it matters**
This is a high-severity isolation failure for OTel deployment patterns that create/recreate multiple Beat receivers in-process: concurrent construction can trigger `fatal error: concurrent map writes` or race-detector failures, crashing Collector startup/reload.

**Suggested fix**
1. Make filebeat processor registration idempotent and one-time (e.g. `sync.Once` around `include.InitializeModule` registration side effects), and
2. Make `libbeat/processors.Namespace` registration path thread-safe (mutex around `Register/add`), so concurrent receiver construction cannot corrupt global maps.

## Reproduction test sketch

```go
func TestConcurrentFilebeatReceiverConstruction(t *testing.T) {
// Build two filebeatreceiver instances concurrently from the same process.
// Run with: go test -race ./x-pack/filebeat/fbreceiver -run TestConcurrentFilebeatReceiverConstruction -count=1
// Expected current behavior: race on libbeat/processors Namespace.reg map (and possible concurrent map writes panic).
}
```

## Safe / already-tracked context

- Existing open issue `#50029` tracks a different concurrent map-write path (`add_nomad_metadata` default maps).
- Existing open issues `#49610` and `#49877` track other process-global receiver-construction state.
- This report is specifically for the distinct processor registry map race in `libbeat/processors/namespace.go` during receiver initialization.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: OTel BeatReceiver Global State Isolation](https://github.com/elastic/beats/actions/runs/26945949290)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 11, 2026, 10:49 AM UTC

Contributor guide

Open the contributing guide

Research direction

Start with x-pack/filebeat/fbreceiver/factory.go and the initializer path through x-pack/filebeat/include/list.go, then inspect libbeat/processors/namespace.go and its registry callers. Add the focused concurrent construction test described in the issue and run go test -race ./x-pack/filebeat/fbreceiver -run TestConcurrentFilebeatReceiverConstruction -count=1. Done means concurrent construction no longer reports registry races or map-write failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.