elastic / elastic/integrations

[pipeline-error-handling] HAProxy log pipeline fails to classify ingest failures as pipeline_error

Open
#18,580 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Findings

### 1. HAProxy `log` ingest pipeline has incomplete top-level `on_failure`

**Location**
- `packages/haproxy/data_stream/log/elasticsearch/ingest_pipeline/default.yml:211-214`
- Related context: `packages/haproxy/data_stream/log/elasticsearch/ingest_pipeline/default.yml:13-40` (grok-heavy parsing), `:158-160` (`event.kind` is set to `event` in normal flow)

**Evidence**
```yaml
on_failure:
- set:
field: error.message
value: '\{\{ _ingest.on_failure_message }}'
```

The failure handler sets `error.message` but does not set `event.kind: pipeline_error`.

**What is wrong**
When parsing fails, the pipeline does not classify the document as a pipeline failure event. In this pipeline, normal processing sets `event.kind: event` (`:158-160`), so failure-path documents are not explicitly marked as `pipeline_error`.

**Why it matters**
`haproxy/log` is a free-form log source with multiple grok patterns (`:13-40`), where parse failures are expected in real deployments. Without `event.kind: pipeline_error`, failures are much harder to discover and triage at scale, even when `error.message` exists.

**Suggested fix**
Add `event.kind: pipeline_error` to the top-level `on_failure` block while keeping the existing `error.message` assignment, for example:

```yaml
on_failure:
- set:
field: event.kind
value: pipeline_error
- set:
field: error.message
value: '\{\{ _ingest.on_failure_message }}'
```

## Scan summary

- Pipelines scanned: **1667**
- Missing `on_failure`: **207**
- Incomplete `on_failure` (missing `event.kind: pipeline_error` and/or `error.message`): **212**

## Notes on de-duplication

- Existing open tracker `#18026` already covers `system/syslog`, `vsphere/log`, and `sonicwall_firewall` preserve behavior, so those were intentionally excluded from this report.

## Suggested actions

- [ ] Add `event.kind: pipeline_error` to `packages/haproxy/data_stream/log/elasticsearch/ingest_pipeline/default.yml` top-level `on_failure`.
- [ ] Add a regression test/assertion to ensure HAProxy ingest failures are emitted with `event.kind: pipeline_error`.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Pipeline Error Handler Compliance](https://github.com/elastic/integrations/actions/runs/24770882634)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Apr 29, 2026, 9:48 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.