elastic / elastic/integrations
[pipeline-error-handling] Logstash log/slowlog pipelines miss pipeline_error classification on failures
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Scan summary
- Pipelines scanned: **1677** (`packages/*/data_stream/*/elasticsearch/ingest_pipeline/*.yml`)
- Missing top-level `on_failure`: **207**
- Incomplete top-level `on_failure` (missing `event.kind: pipeline_error` and/or `error.message`): **214**
## Findings
### 1. Logstash `log` and `slowlog` pipeline failures are not classified as `pipeline_error`
**Severity:** High
**Locations (verified):**
- `packages/logstash/data_stream/log/elasticsearch/ingest_pipeline/default.yml:25-28`
- `packages/logstash/data_stream/log/elasticsearch/ingest_pipeline/pipeline-plaintext.yml:3-6`
- `packages/logstash/data_stream/log/elasticsearch/ingest_pipeline/pipeline-json.yml:3-6`
- `packages/logstash/data_stream/slowlog/elasticsearch/ingest_pipeline/default.yml:25-28`
- `packages/logstash/data_stream/slowlog/elasticsearch/ingest_pipeline/pipeline-plaintext.yml:3-6`
- `packages/logstash/data_stream/slowlog/elasticsearch/ingest_pipeline/pipeline-json.yml:3-6`
**Evidence:**
Current top-level and branch pipeline `on_failure` handlers set only `error.message`, for example:
```yaml
on_failure:
- set:
field: error.message
value: '\{\{ _ingest.on_failure_message }}'
```
Normal success paths explicitly set `event.kind` to `event` in format-specific pipelines:
- `packages/logstash/data_stream/log/elasticsearch/ingest_pipeline/pipeline-plaintext.yml:47`
- `packages/logstash/data_stream/log/elasticsearch/ingest_pipeline/pipeline-json.yml:46`
- `packages/logstash/data_stream/slowlog/elasticsearch/ingest_pipeline/pipeline-plaintext.yml:57`
- `packages/logstash/data_stream/slowlog/elasticsearch/ingest_pipeline/pipeline-json.yml:47`
No corresponding failure-path `event.kind: pipeline_error` is set.
**What is wrong:**
Parse failures are recorded as text in `error.message` but not classified as pipeline failures.
**Why it matters:**
These datasets process free-form plaintext and JSON Logstash logs using grok/json parsing. Parse failures are realistic in production (format drift, truncation, unexpected lines). Without `event.kind: pipeline_error`, failure documents are harder to discover with standard pipeline-error filters and alerts.
**Regression check:**
`git log -p` for both `default.yml` files shows this omission has existed since pipeline introduction (commit `59f0b7f42e`), with no later fix adding failure-path classification.
## `preserve_original_event` review
Packages exposing `preserve_original_event` were reviewed (`azure_blob_storage`, `google_cloud_storage`, `http_endpoint`, `sonicwall_firewall`, `tcp`, `udp`, `winlog`). No new high-severity preserve gap was confirmed in this run.
## Highest-priority missing-handler review
No new high-priority missing top-level `on_failure` case was confirmed in this run beyond already tracked open issues.
## Suggested actions
- [ ] Add `event.kind: pipeline_error` to all six listed `on_failure` handlers in Logstash `log`/`slowlog` pipelines while keeping existing `error.message` assignment.
- [ ] Add ingest-pipeline tests that assert failure-path documents include both `error.message` and `event.kind: pipeline_error`.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Pipeline Error Handler Compliance](https://github.com/elastic/integrations/actions/runs/26155633808)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 27, 2026, 10:17 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.