elastic / elastic/elastic-package
System tests reject pipeline_error documents that legitimately carry error.message
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
## Problem
`validateFields` unconditionally fails any system test that produces a document with `error.message` set:
```go
// internal/testrunner/runners/system/tester.go
if message, err := doc.GetValue("error.message"); err != common.ErrKeyNotFound {
multiErr = append(multiErr, fmt.Errorf("found error.message in event: %v", message))
continue
}
```
This is the right check for most cases — an unexpected `error.message` on a normal event usually means the ingest pipeline's `on_failure` handler ran when it shouldn't have. But it incorrectly rejects **intentional** error events that are correctly formed per the ECS spec.
## ECS context
`event.kind: pipeline_error` is the ECS-defined value for an event that represents a processing error or failure. Ingest pipelines that handle error paths properly set this value explicitly. A document with both `event.kind: pipeline_error` and `error.message` is not broken — it is a correctly-formed, intentional error event.
The current check conflates these two distinct cases:
- `event.kind: metric` + `error.message` → ingest pipeline failed unexpectedly (should fail the test)
- `event.kind: pipeline_error` + `error.message` → intentional error event, valid by ECS (should not fail the test)
## Concrete example
The `mongodb_atlas` integration fixes a state-preservation bug: when the Atlas measurements endpoint returns HTTP 503, the CEL program must preserve `group_id` in state so the next polling interval can continue (elastic/integrations#20226). To test this, system tests point a mock server at a group ID that always returns 503 for measurement requests.
The CEL program correctly emits:
```json
{ "error": { "code": "SERVICE_UNAVAILABLE", "message": "mock: measurements unavailable" } }
```
The ingest pipeline handles this correctly — it sets `event.kind: pipeline_error` because `error.message` is present. The document is well-formed and maps cleanly. But `validateFields` rejects it, so the tests always fail.
These tests had to be removed entirely, leaving a gap in automated regression coverage for a real customer-reported bug. There is no config option to work around this.
## Proposed fix
Skip the `error.message` check for documents where `event.kind` is `"pipeline_error"`:
```go
if message, err := doc.GetValue("error.message"); err != common.ErrKeyNotFound {
if kind, _ := doc.GetValue("event.kind"); kind == "pipeline_error" {
// Intentional error event — event.kind: pipeline_error is the ECS-blessed
// signal for this. Proceed to normal field validation instead.
} else {
multiErr = append(multiErr, fmt.Errorf("found error.message in event: %v", message))
continue
}
}
```
This is a semantic correctness fix, not a per-test escape hatch. No new config option is needed. The strict behavior (fail on unexpected `error.message`) is preserved for all documents where `event.kind` is anything other than `pipeline_error`.
If some package authors want to enforce that their tests produce zero `pipeline_error` documents even when they are technically valid, a follow-up `assert.disallow_pipeline_errors: true` config option could cover that case.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in internal/testrunner/runners/system/tester.go and inspect validateFields, then run the affected system tests around the mongodb_atlas 503 scenario. Done means documents with event.kind set to pipeline_error can retain error.message, while unexpected error.message values on other events still fail validation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100