elastic / elastic/elastic-package

System tests reject pipeline_error documents that legitimately carry error.message

Open Beginner friendly
#3,772 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

## Problem

`validateFields` unconditionally fails any system test that produces a document with `error.message` set:

```go
// internal/testrunner/runners/system/tester.go
if message, err := doc.GetValue("error.message"); err != common.ErrKeyNotFound {
multiErr = append(multiErr, fmt.Errorf("found error.message in event: %v", message))
continue
}
```

This is the right check for most cases — an unexpected `error.message` on a normal event usually means the ingest pipeline's `on_failure` handler ran when it shouldn't have. But it incorrectly rejects **intentional** error events that are correctly formed per the ECS spec.

## ECS context

`event.kind: pipeline_error` is the ECS-defined value for an event that represents a processing error or failure. Ingest pipelines that handle error paths properly set this value explicitly. A document with both `event.kind: pipeline_error` and `error.message` is not broken — it is a correctly-formed, intentional error event.

The current check conflates these two distinct cases:
- `event.kind: metric` + `error.message` → ingest pipeline failed unexpectedly (should fail the test)
- `event.kind: pipeline_error` + `error.message` → intentional error event, valid by ECS (should not fail the test)

## Concrete example

The `mongodb_atlas` integration fixes a state-preservation bug: when the Atlas measurements endpoint returns HTTP 503, the CEL program must preserve `group_id` in state so the next polling interval can continue (elastic/integrations#20226). To test this, system tests point a mock server at a group ID that always returns 503 for measurement requests.

The CEL program correctly emits:
```json
{ "error": { "code": "SERVICE_UNAVAILABLE", "message": "mock: measurements unavailable" } }
```

The ingest pipeline handles this correctly — it sets `event.kind: pipeline_error` because `error.message` is present. The document is well-formed and maps cleanly. But `validateFields` rejects it, so the tests always fail.

These tests had to be removed entirely, leaving a gap in automated regression coverage for a real customer-reported bug. There is no config option to work around this.

## Proposed fix

Skip the `error.message` check for documents where `event.kind` is `"pipeline_error"`:

```go
if message, err := doc.GetValue("error.message"); err != common.ErrKeyNotFound {
if kind, _ := doc.GetValue("event.kind"); kind == "pipeline_error" {
// Intentional error event — event.kind: pipeline_error is the ECS-blessed
// signal for this. Proceed to normal field validation instead.
} else {
multiErr = append(multiErr, fmt.Errorf("found error.message in event: %v", message))
continue
}
}
```

This is a semantic correctness fix, not a per-test escape hatch. No new config option is needed. The strict behavior (fail on unexpected `error.message`) is preserved for all documents where `event.kind` is anything other than `pipeline_error`.

If some package authors want to enforce that their tests produce zero `pipeline_error` documents even when they are technically valid, a follow-up `assert.disallow_pipeline_errors: true` config option could cover that case.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/testrunner/runners/system/tester.go and inspect validateFields, then run the affected system tests around the mongodb_atlas 503 scenario. Done means documents with event.kind set to pipeline_error can retain error.message, while unexpected error.message values on other events still fail validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.