[bug-hunter] add_kubernetes_metadata Close hangs when ServerVersion call blocks
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Impact
`add_kubernetes_metadata` can hang Beat shutdown indefinitely when Kubernetes API version detection blocks instead of returning an error. This prevents clean restarts/rollouts and leaves the process stuck in `Close()`.
## Reproduction Steps
1. Create this test file: `libbeat/processors/add_kubernetes_metadata/kubernetes_blocking_repro_test.go`
```go
package add_kubernetes_metadata
import (
"context"
"testing"
"time"
"k8s.io/apimachinery/pkg/runtime"
k8stesting "k8s.io/client-go/testing"
k8sfake "k8s.io/client-go/kubernetes/fake"
k8sclient "k8s.io/client-go/kubernetes"
"github.com/elastic/elastic-agent-libs/logp/logptest"
)
func blockingK8sClient() k8sclient.Interface {
client := k8sfake.NewSimpleClientset()
client.Fake.PrependReactor("get", "version", func(k8stesting.Action) (bool, runtime.Object, error) {
select {}
})
return client
}
func TestCloseDoesNotBlockWhenServerVersionCallBlocks(t *testing.T) {
logger := logptest.NewTestingLogger(t, selector)
ctx, cancel := context.WithCancel(context.Background())
proc := &kubernetesAnnotator{
log: logger,
cache: newCache(10 * time.Second),
cancelCtx: cancel,
}
proc.wg.Add(1)
initStarted := make(chan struct{})
go func() {
defer proc.wg.Done()
proc.initOnce.Do(func() {
close(initStarted)
_, _ = isKubernetesAvailableWithTimeout(
ctx,
blockingK8sClient(),
0,
10*time.Millisecond,
logger,
)
})
}()
<-initStarted
closeDone := make(chan struct{})
go func() {
_ = proc.Close()
close(closeDone)
}()
select {
case <-closeDone:
case <-time.After(500 * time.Millisecond):
t.Fatal("Close blocked while waiting for a stuck kubernetes ServerVersion call")
}
}
```
2. Run:
```bash
go test -run TestCloseDoesNotBlockWhenServerVersionCallBlocks -count=1 ./libbeat/processors/add_kubernetes_metadata
```
## Expected vs Actual
**Expected:** `Close()` should return promptly after cancellation, even if Kubernetes availability probing is waiting.
**Actual:** test fails because `Close()` remains blocked:
```text
--- FAIL: TestCloseDoesNotBlockWhenServerVersionCallBlocks (0.50s)
kubernetes_blocking_repro_test.go:65: Close blocked while waiting for a stuck kubernetes ServerVersion call
FAIL
FAIL github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata 0.511s
FAIL
```
## Failing Test
See the full reproduction test above.
## Evidence
- `libbeat/processors/add_kubernetes_metadata/kubernetes.go:105` calls `isKubernetesAvailable(client, logger)` directly inside the loop.
- `libbeat/processors/add_kubernetes_metadata/kubernetes.go:110-117` checks cancellation/timeout **after** that call returns.
- `libbeat/processors/add_kubernetes_metadata/kubernetes.go:429-433` shows `Close()` calls `cancelCtx()` and then blocks on `k.wg.Wait()`.
So when `ServerVersion()` blocks, cancellation cannot be observed and `Close()` waits indefinitely.
> [!NOTE]
>
> 🔒 Integrity filter blocked 2 items
>
> The following items were blocked because they don't meet the GitHub integrity level.
>
> - [#50509](https://github.com/elastic/beats/pull/50509) `search_pull_requests`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
> - [#50507](https://github.com/elastic/beats/issues/50507) `issue_read`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Bug Hunter](https://github.com/elastic/beats/actions/runs/27204663657)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 16, 2026, 12:14 PM UTC
Contributor guide
Assessment
This issue has not been assessed yet.