elastic / elastic/elastic-package

system_test_assert_conditions/test/min_count flaky: Elasticsearch returns `minDoc must be >= 0` 400 error

Open
#3,877 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

## Summary

The `system_test_assert_conditions/test/min_count` system test case fails intermittently with a 400 Bad Request from Elasticsearch. The failure is non-deterministic (different negative `minDoc` values each time) and unrelated to code changes — it has been observed on `main` builds and unrelated PRs.

## Error

```
FAIL: system_test_assert_conditions/test/min_count (system)
error: failed to search docs for data stream logs-system_test_assert_conditions.test-:
[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception",
"reason":"minDoc must be >= 0 but got minDoc=-500"}], ...}}
```

## Affected builds

| Build | Branch | Date | minDoc value |
|-------|--------|------|-------------|
| [#8591](https://buildkite.com/elastic/elastic-package/builds/8591) | `main` | 2026-08-25 | `-423` |
| [#8597](https://buildkite.com/elastic/elastic-package/builds/8597) | `main` | 2026-08-27 | `-261` |
| [#8603](https://buildkite.com/elastic/elastic-package/builds/8603) | `fix/terraform-deployer-gcloud-cli` | 2026-08-27 | `-500` |

## Root cause

`getDocs` in `internal/testrunner/runners/system/tester.go` always queries with `WithSize(500)` + `WithSort("@timestamp:asc")`. When Elasticsearch is still indexing and a shard is in a transitional state with fewer than 500 documents in a segment, the sort collector's internal `lastDoc - numDocs` calculation goes negative, producing the 400.

The 400 is treated as a fatal error and kills the test immediately — it is never retried — even though it is a transient condition.

A contributing factor: the test config `data_stream/test/_dev/test/system/test-min_count-config.yml` sets `assert.min_count: 500`, which is exactly `elasticsearchQuerySize = 500` (tester.go:125). The test intentionally probes this boundary (a comment in the config notes this), but doing so is what pushes the search into the range where the ES behaviour triggers.

## Proposed fix

Two complementary changes:

**1. `internal/testrunner/runners/system/tester.go` — handle the transient 400 gracefully**

Mirror the existing pattern for `no_shard_available_action_exception` (tester.go:853): detect the `minDoc must be >= 0` 400 and return `(&hits{}, nil)` instead of a fatal error, allowing the `UntilTrue` retry loop to wait for the shard to stabilise.

```go
if resp.StatusCode == http.StatusBadRequest &&
strings.Contains(resp.String(), "minDoc must be >= 0") {
return &hits{}, nil
}
```

**2. `test-min_count-config.yml` — reduce `min_count` away from the query size boundary**

`min_count: 500` == `elasticsearchQuerySize` is the exact boundary that triggers the issue. Reducing to e.g. `50` (still exercises the `min_count` assertion feature) removes the boundary condition entirely.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/testrunner/runners/system/tester.go at getDocs and compare its Elasticsearch error handling with the existing no_shard_available_action_exception pattern. Review data_stream/test/_dev/test/system/test-min_count-config.yml, then run system_test_assert_conditions/test/min_count. Done means the transient minDoc 400 is retried and the test no longer probes the query-size boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.