elastic / elastic/elastic-package
system_test_assert_conditions/test/min_count flaky: Elasticsearch returns `minDoc must be >= 0` 400 error
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
## Summary
The `system_test_assert_conditions/test/min_count` system test case fails intermittently with a 400 Bad Request from Elasticsearch. The failure is non-deterministic (different negative `minDoc` values each time) and unrelated to code changes — it has been observed on `main` builds and unrelated PRs.
## Error
```
FAIL: system_test_assert_conditions/test/min_count (system)
error: failed to search docs for data stream logs-system_test_assert_conditions.test-:
[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception",
"reason":"minDoc must be >= 0 but got minDoc=-500"}], ...}}
```
## Affected builds
| Build | Branch | Date | minDoc value |
|-------|--------|------|-------------|
| [#8591](https://buildkite.com/elastic/elastic-package/builds/8591) | `main` | 2026-08-25 | `-423` |
| [#8597](https://buildkite.com/elastic/elastic-package/builds/8597) | `main` | 2026-08-27 | `-261` |
| [#8603](https://buildkite.com/elastic/elastic-package/builds/8603) | `fix/terraform-deployer-gcloud-cli` | 2026-08-27 | `-500` |
## Root cause
`getDocs` in `internal/testrunner/runners/system/tester.go` always queries with `WithSize(500)` + `WithSort("@timestamp:asc")`. When Elasticsearch is still indexing and a shard is in a transitional state with fewer than 500 documents in a segment, the sort collector's internal `lastDoc - numDocs` calculation goes negative, producing the 400.
The 400 is treated as a fatal error and kills the test immediately — it is never retried — even though it is a transient condition.
A contributing factor: the test config `data_stream/test/_dev/test/system/test-min_count-config.yml` sets `assert.min_count: 500`, which is exactly `elasticsearchQuerySize = 500` (tester.go:125). The test intentionally probes this boundary (a comment in the config notes this), but doing so is what pushes the search into the range where the ES behaviour triggers.
## Proposed fix
Two complementary changes:
**1. `internal/testrunner/runners/system/tester.go` — handle the transient 400 gracefully**
Mirror the existing pattern for `no_shard_available_action_exception` (tester.go:853): detect the `minDoc must be >= 0` 400 and return `(&hits{}, nil)` instead of a fatal error, allowing the `UntilTrue` retry loop to wait for the shard to stabilise.
```go
if resp.StatusCode == http.StatusBadRequest &&
strings.Contains(resp.String(), "minDoc must be >= 0") {
return &hits{}, nil
}
```
**2. `test-min_count-config.yml` — reduce `min_count` away from the query size boundary**
`min_count: 500` == `elasticsearchQuerySize` is the exact boundary that triggers the issue. Reducing to e.g. `50` (still exercises the `min_count` assertion feature) removes the boundary condition entirely.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in internal/testrunner/runners/system/tester.go at getDocs and compare its Elasticsearch error handling with the existing no_shard_available_action_exception pattern. Review data_stream/test/_dev/test/system/test-min_count-config.yml, then run system_test_assert_conditions/test/min_count. Done means the transient minDoc 400 is retried and the test no longer probes the query-size boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100