[Logstash - Investigation] Concurrent `PipelineAction::Create` at boot can raise `MetricStore` `KeyError: key not found`
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
When several pipelines start concurrently, one pipeline's `Create` intermittently fails with `KeyError: key not found` while registering flow metrics. Evidence points to a race condition in metric registration, needs investigation to confirm and to implement the proper fix.
## Evidence
- Backtrace (key frames):
```
KeyError: key not found
concurrent/map.rb:in `raise_fetch_no_key' / `fetch' / `fetch_or_store'
logstash/instrument/metric_store.rb:in `block in fetch_or_store'
logstash/instrument/collector.rb:in `register?'
org/logstash/execution/AbstractPipelineExt.java:in `initialize_flow_metrics'
logstash/agent.rb:in `block in converge_state'
```
- **Intermittent** (a different pipeline each time). Signature of a timing race, not a config/compile error.
- At boot each `Create` runs on its pwm, so many threads hit the single process-wide collector at once.
## Code context
The random, intermittent pipeline error looks like a race condition, but the raised exception comes from a call protected by `@structured_lookup_mutex`, so I think every structured-store write is serialized. A call serialized that way would behave single-threaded and normally never raise. Resolving that contradiction is the goal of this issue.
- The `fetch_or_store` call in Logstash that raises — [metric_store.rb#L85](https://github.com/elastic/logstash/blob/main/logstash-core/lib/logstash/instrument/metric_store.rb#L85)
- Where it throws in `concurrent-ruby` (`~> 1.3`) — [`raise_fetch_no_key` map.rb#L327](https://github.com/ruby-concurrency/concurrent-ruby/blob/v1.3.5/lib/concurrent-ruby/concurrent/map.rb#L327)
Contributor guide
Research direction
Start at logstash-core/lib/logstash/instrument/metric_store.rb#L85 and trace the synchronized fetch_or_store call through collector.rb#register? and concurrent-ruby's map.rb#L327. Reproduce or inspect concurrent pipeline boot through AbstractPipelineExt#initialize_flow_metrics and agent.rb#converge_state, focusing on the reported mutex contradiction. Done means the race cause is confirmed and the appropriate fix prevents the intermittent KeyError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, ruby
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100