elastic / elastic/logstash

[Logstash - Investigation] Concurrent `PipelineAction::Create` at boot can raise `MetricStore` `KeyError: key not found`

Open
#19,489 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

When several pipelines start concurrently, one pipeline's `Create` intermittently fails with `KeyError: key not found` while registering flow metrics. Evidence points to a race condition in metric registration, needs investigation to confirm and to implement the proper fix.

## Evidence
- Backtrace (key frames):
```
KeyError: key not found
concurrent/map.rb:in `raise_fetch_no_key' / `fetch' / `fetch_or_store'
logstash/instrument/metric_store.rb:in `block in fetch_or_store'
logstash/instrument/collector.rb:in `register?'
org/logstash/execution/AbstractPipelineExt.java:in `initialize_flow_metrics'
logstash/agent.rb:in `block in converge_state'
```
- **Intermittent** (a different pipeline each time). Signature of a timing race, not a config/compile error.
- At boot each `Create` runs on its pwm, so many threads hit the single process-wide collector at once.

## Code context

The random, intermittent pipeline error looks like a race condition, but the raised exception comes from a call protected by `@structured_lookup_mutex`, so I think every structured-store write is serialized. A call serialized that way would behave single-threaded and normally never raise. Resolving that contradiction is the goal of this issue.

- The `fetch_or_store` call in Logstash that raises — [metric_store.rb#L85](https://github.com/elastic/logstash/blob/main/logstash-core/lib/logstash/instrument/metric_store.rb#L85)
- Where it throws in `concurrent-ruby` (`~> 1.3`) — [`raise_fetch_no_key` map.rb#L327](https://github.com/ruby-concurrency/concurrent-ruby/blob/v1.3.5/lib/concurrent-ruby/concurrent/map.rb#L327)

Contributor guide

Open the contributing guide

Research direction

Start at logstash-core/lib/logstash/instrument/metric_store.rb#L85 and trace the synchronized fetch_or_store call through collector.rb#register? and concurrent-ruby's map.rb#L327. Reproduce or inspect concurrent pipeline boot through AbstractPipelineExt#initialize_flow_metrics and agent.rb#converge_state, focusing on the reported mutex contradiction. Done means the race cause is confirmed and the appropriate fix prevents the intermittent KeyError.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, ruby
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.