elastic / elastic/beats

[bug-hunter] Filebeat shared ES state store uses first context and cancels sibling instances

Open
#49,804 1 comment 0 reactions 1 assignee Claimed by @orestisfl View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

## Impact
When Filebeat runs multiple beater instances that share the same registry path and Elasticsearch state-store backend, registry lifecycle is tied to the first instance context. Canceling the first instance can break state-store operations in still-running sibling instances, causing deterministic `context canceled` failures.

## Reproduction Steps
1. In `filebeat/beater`, create this new test file:

```go
package beater

import (
"context"
"testing"

"github.com/stretchr/testify/require"

"github.com/elastic/beats/v7/filebeat/config"
"github.com/elastic/beats/v7/filebeat/features"
"github.com/elastic/beats/v7/libbeat/beat"
"github.com/elastic/elastic-agent-libs/logp"
"github.com/elastic/elastic-agent-libs/paths"
)

func TestOpenStateStore_ESSharedRegistryUsesFirstContext(t *testing.T) {
t.Setenv("AGENTLESS_ELASTICSEARCH_STATE_STORE_INPUT_TYPES", "test")
features.ReinitForTest()
t.Cleanup(func() {
t.Setenv("AGENTLESS_ELASTICSEARCH_STATE_STORE_INPUT_TYPES", "")
features.ReinitForTest()
})

beatPaths := paths.New()
beatPaths.Data = t.TempDir()

cfg := config.Registry{Path: "", Permissions: 0600}

ctx1, cancel1 := context.WithCancel(context.Background())
_, err := openStateStore(ctx1, beat.Info{Beat: "test"}, logp.NewNopLogger(), cfg, beatPaths)
require.NoError(t, err)

ctx2, cancel2 := context.WithCancel(context.Background())
defer cancel2()
s2, err := openStateStore(ctx2, beat.Info{Beat: "test"}, logp.NewNopLogger(), cfg, beatPaths)
require.NoError(t, err)

store2, err := s2.StoreFor("test")
require.NoError(t, err)

done := make(chan error, 1)
go func() {
_, err := store2.Has("k")
done <- err
}()

cancel1()

err = <-done
if err != nil {
t.Fatalf("store operation returned after canceling first context: %v", err)
}
}
```

2. Run:

```bash
go test -v ./filebeat/beater -run TestOpenStateStore_ESSharedRegistryUsesFirstContext -count=1 -timeout=20s
```

## Expected vs Actual
**Expected:** Canceling the first `openStateStore` context should not affect operations from the second store instance with its own context.

**Actual:** The second store operation fails immediately:

```text
=== RUN TestOpenStateStore_ESSharedRegistryUsesFirstContext
store_repro_test.go:58: store operation returned after canceling the first context: failed in store/has operation on store 'test': context canceled
--- FAIL: TestOpenStateStore_ESSharedRegistryUsesFirstContext (0.00s)
FAIL
```

## Failing Test
The test above is the minimal failing test.

## Evidence
- Shared registry key only uses backend+path, so instances with same path/backend share one registry: `filebeat/beater/store.go:64-69`, `filebeat/beater/store.go:72-79`, `filebeat/beater/store.go:121`.
- Elasticsearch registry is constructed once with the first caller context: `filebeat/beater/store.go:108-116`.
- Later callers reuse the same shared registry: `filebeat/beater/store.go:78-79`, `filebeat/beater/store.go:126-132`.
- ES backend store operations fail when that context is canceled (`waitReady`): `libbeat/statestore/backend/es/store.go:82-88`, `libbeat/statestore/backend/es/store.go:120-123`.
- The change introducing shared registry behavior is from commit `d634b972a8`.

I checked `/tmp/previous-findings.json`; this bug is not listed there.

> [!NOTE]
>
> 🔒 Integrity filtering filtered 1 item
>
> Integrity filtering activated and filtered the following item during workflow execution.
> This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.
>
> - issue:elastic/beats#unknown (`search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/beats/actions/runs/23794590148)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 7, 2026, 11:44 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.