[bug-hunter] Filebeat shared ES state store uses first context and cancels sibling instances
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Impact
When Filebeat runs multiple beater instances that share the same registry path and Elasticsearch state-store backend, registry lifecycle is tied to the first instance context. Canceling the first instance can break state-store operations in still-running sibling instances, causing deterministic `context canceled` failures.
## Reproduction Steps
1. In `filebeat/beater`, create this new test file:
```go
package beater
import (
"context"
"testing"
"github.com/stretchr/testify/require"
"github.com/elastic/beats/v7/filebeat/config"
"github.com/elastic/beats/v7/filebeat/features"
"github.com/elastic/beats/v7/libbeat/beat"
"github.com/elastic/elastic-agent-libs/logp"
"github.com/elastic/elastic-agent-libs/paths"
)
func TestOpenStateStore_ESSharedRegistryUsesFirstContext(t *testing.T) {
t.Setenv("AGENTLESS_ELASTICSEARCH_STATE_STORE_INPUT_TYPES", "test")
features.ReinitForTest()
t.Cleanup(func() {
t.Setenv("AGENTLESS_ELASTICSEARCH_STATE_STORE_INPUT_TYPES", "")
features.ReinitForTest()
})
beatPaths := paths.New()
beatPaths.Data = t.TempDir()
cfg := config.Registry{Path: "", Permissions: 0600}
ctx1, cancel1 := context.WithCancel(context.Background())
_, err := openStateStore(ctx1, beat.Info{Beat: "test"}, logp.NewNopLogger(), cfg, beatPaths)
require.NoError(t, err)
ctx2, cancel2 := context.WithCancel(context.Background())
defer cancel2()
s2, err := openStateStore(ctx2, beat.Info{Beat: "test"}, logp.NewNopLogger(), cfg, beatPaths)
require.NoError(t, err)
store2, err := s2.StoreFor("test")
require.NoError(t, err)
done := make(chan error, 1)
go func() {
_, err := store2.Has("k")
done <- err
}()
cancel1()
err = <-done
if err != nil {
t.Fatalf("store operation returned after canceling first context: %v", err)
}
}
```
2. Run:
```bash
go test -v ./filebeat/beater -run TestOpenStateStore_ESSharedRegistryUsesFirstContext -count=1 -timeout=20s
```
## Expected vs Actual
**Expected:** Canceling the first `openStateStore` context should not affect operations from the second store instance with its own context.
**Actual:** The second store operation fails immediately:
```text
=== RUN TestOpenStateStore_ESSharedRegistryUsesFirstContext
store_repro_test.go:58: store operation returned after canceling the first context: failed in store/has operation on store 'test': context canceled
--- FAIL: TestOpenStateStore_ESSharedRegistryUsesFirstContext (0.00s)
FAIL
```
## Failing Test
The test above is the minimal failing test.
## Evidence
- Shared registry key only uses backend+path, so instances with same path/backend share one registry: `filebeat/beater/store.go:64-69`, `filebeat/beater/store.go:72-79`, `filebeat/beater/store.go:121`.
- Elasticsearch registry is constructed once with the first caller context: `filebeat/beater/store.go:108-116`.
- Later callers reuse the same shared registry: `filebeat/beater/store.go:78-79`, `filebeat/beater/store.go:126-132`.
- ES backend store operations fail when that context is canceled (`waitReady`): `libbeat/statestore/backend/es/store.go:82-88`, `libbeat/statestore/backend/es/store.go:120-123`.
- The change introducing shared registry behavior is from commit `d634b972a8`.
I checked `/tmp/previous-findings.json`; this bug is not listed there.
> [!NOTE]
>
> 🔒 Integrity filtering filtered 1 item
>
> Integrity filtering activated and filtered the following item during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - issue:elastic/beats#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/beats/actions/runs/23794590148)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 7, 2026, 11:44 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.