elastic / elastic/beats

[Flaky Test] libbeat/api: Log in goroutine after TestHTTP has completed:

Open
#44,771 1 comment 0 reactions 0 assignees View on GitHub
flaky-test Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Flaky Test

* **Test Name:** libbeat/api
* **Link:** https://github.com/elastic/beats/tree/main/libbeat/api
* **Branch:** main, AndersonQ:42762-dropped-event-per-input
* **Artifact Link:** [beats-libbeat_build_16509_ubuntu-libbeat-go-integration-tests.log.txt](https://github.com/user-attachments/files/20705707/beats-libbeat_build_16509_ubuntu-libbeat-go-integration-tests.log.txt)

* **Notes:** Additional details about the test. e.g. theory as to failure cause

It's a known Go issue that can be avoided: https://github.com/golang/go/issues/30389
It's easy to reproduce it ~(on my machine)~ with
```
go test -count 20000 ./libbeat/api

panic: Log in goroutine after TestHTTP has completed: 2025-06-12T11:30:47.151+0200 INFO api logp/logger.go:206 Stats endpoint (127.0.0.1:33591) finished: accept tcp 127.0.0.1:33591: use of closed network connection

goroutine 475761 [running]:
testing.(*common).logDepth(0xc063f40fc0, {0xc0632726e0, 0xa0}, 0x3)
/home/ainsoph/devel/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.linux-amd64/src/testing/testing.go:1064 +0x4b4
testing.(*common).log(...)
/home/ainsoph/devel/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.linux-amd64/src/testing/testing.go:1046
testing.(*common).Logf(0xc063f40fc0, {0x8c5036?, 0x400?}, {0xc04cab6240?, 0x795811?, 0x3?})
/home/ainsoph/devel/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.3.linux-amd64/src/testing/testing.go:1097 +0x54
go.uber.org/zap/zaptest.TestingWriter.Write({{0x7f99c9aad2c8?, 0xc063f40fc0?}, 0xc0?}, {0xc04ca6c800?, 0xa1, 0xc04ca6c800?})
/home/ainsoph/devel/go/pkg/mod/go.uber.org/zap@v1.27.0/zaptest/logger.go:146 +0xdb
go.uber.org/zap/zapcore.(*ioCore).Write(0xc04ce250e0, {0x0, {0xc20ac571c906f4ba, 0x6d992bf72, 0xc52e00}, {0x8c517e, 0x3}, {0xc04d09e8c0, 0x67}, {0x1, ...}, ...}, ...)
/home/ainsoph/devel/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0xb5
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc03c291d40, {0x0, 0x0, 0x0})
/home/ainsoph/devel/go/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x119
go.uber.org/zap.(*SugaredLogger).log(0xc0632689b0, 0x0, {0x8d2d99?, 0x20?}, {0xc062145f70?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
/home/ainsoph/devel/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:355 +0xec
go.uber.org/zap.(*SugaredLogger).Infof(...)
/home/ainsoph/devel/go/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:204
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
/home/ainsoph/devel/go/pkg/mod/github.com/elastic/elastic-agent-libs@v0.19.6/logp/logger.go:206
github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1({0x96a670, 0xc06383cec0})
/home/ainsoph/devel/github.com/elastic/beats/libbeat/api/server.go:69 +0x20c
created by github.com/elastic/beats/v7/libbeat/api.(*Server).Start in goroutine 475769
/home/ainsoph/devel/github.com/elastic/beats/libbeat/api/server.go:66 +0x110
FAIL github.com/elastic/beats/v7/libbeat/api 30.118s
FAIL
```

### Stack Trace

```
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/go/src/testing/testing.go:1064 +0x4b4
2025-06-11 18:23:30 CEST
testing.(*common).log(...)
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/go/src/testing/testing.go:1046
2025-06-11 18:23:30 CEST
testing.(*common).Logf(0xc0000ff880, {0x8e8176?, 0x400?}, {0xc00003d8b0?, 0x7b2f43?, 0x3?})
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/go/src/testing/testing.go:1097 +0x54
2025-06-11 18:23:30 CEST
go.uber.org/zap/zaptest.TestingWriter.Write({{0x71ef9643e310?, 0xc0000ff880?}, 0x80?}, {0xc0001fa000?, 0x9d, 0xc0001fa000?})
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/go.uber.org/zap@v1.27.0/zaptest/logger.go:146 +0xdb
2025-06-11 18:23:30 CEST
go.uber.org/zap/zapcore.(*ioCore).Write(0xc00019ecc0, {0x0, {0xc20a88a792730549, 0x73dcc0, 0xc85f60}, {0x8e82c3, 0x3}, {0xc000035650, 0x67}, {0x1, ...}, ...}, ...)
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/core.go:99 +0xb5
2025-06-11 18:23:30 CEST
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0001a01a0, {0x0, 0x0, 0x0})
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/go.uber.org/zap@v1.27.0/zapcore/entry.go:253 +0x119
2025-06-11 18:23:30 CEST
go.uber.org/zap.(*SugaredLogger).log(0xc00007c298, 0x0, {0x8f64ca?, 0x20?}, {0xc000200f70?, 0x2?, 0x2?}, {0x0, 0x0, 0x0})
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:355 +0xec
2025-06-11 18:23:30 CEST
go.uber.org/zap.(*SugaredLogger).Infof(...)
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/go.uber.org/zap@v1.27.0/sugar.go:204
2025-06-11 18:23:30 CEST
github.com/elastic/elastic-agent-libs/logp.(*Logger).Infof(...)
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/.asdf/installs/golang/1.24.3/packages/pkg/mod/github.com/elastic/elastic-agent-libs@v0.19.6/logp/logger.go:206
2025-06-11 18:23:30 CEST
github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1({0x991510, 0xc0000a56c0})
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1749657779978475881/elastic/beats-libbeat/libbeat/api/server.go:69 +0x21e
2025-06-11 18:23:30 CEST
created by github.com/elastic/beats/v7/libbeat/api.(*Server).Start in goroutine 37
2025-06-11 18:23:30 CEST
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1749657779978475881/elastic/beats-libbeat/libbeat/api/server.go:66 +0x14c
2025-06-11 18:23:30 CEST
FAIL github.com/elastic/beats/v7/libbeat/api 0.018s
```

Contributor guide

Open the contributing guide

Research direction

Start by running `go test -count 20000 ./libbeat/api` and read `libbeat/api/server.go` around the goroutine at lines 66-69, along with the `TestHTTP` setup and teardown. Trace the server shutdown path and verify the test cannot complete while that goroutine can still log; done means the repeated package test finishes without the post-test logging panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.