[pipeline-error-handling] High-impact log pipelines miss pipeline_error classification in on_failure

## Findings summary

I scanned ingest pipelines in this repository and then manually verified high-priority free-form log pipelines.

- Total pipelines scanned: **1700**
- Pipelines with missing `on_failure`: **209**
- Pipelines with incomplete `on_failure` (missing `event.kind` and/or `error.message`): **213**

This issue focuses only on high-impact free-form log sources where parse failures are expected in production.

## High-severity findings

### 1. `system` syslog pipelines do not classify failures as `pipeline_error`

**Locations**
- `packages/system/data_stream/syslog/elasticsearch/ingest_pipeline/default.yml:4-11,28-34`
- `packages/system/data_stream/syslog/elasticsearch/ingest_pipeline/log.yml:46-52`
- `packages/system/data_stream/syslog/elasticsearch/ingest_pipeline/journald.yml:36-42`

**Evidence**
- Entry parser is free-form grok on syslog message:
- `default.yml:4-11` (multiple syslog grok patterns)
- Top-level failure handler only sets `error.message`:
- `default.yml:28-34`
- Child pipelines `log.yml` and `journald.yml` also only set `error.message` in `on_failure`.

**What is wrong**
`on_failure` does not set `event.kind: pipeline_error`, so failed parses are not explicitly classified as pipeline failures.

**Why it matters**
`system` syslog is a high-volume, free-form source with inevitable format variance. Without `event.kind: pipeline_error`, operators cannot reliably find parse-failed documents via standard failure queries/alerts.

**Suggested fix**
Add to `on_failure` in the affected pipelines:

```yaml
- set:
field: event.kind
value: pipeline_error
```

while retaining existing `error.message` population.

### 2. `nginx` access pipeline `on_failure` lacks `event.kind: pipeline_error`

**Location**
- `packages/nginx/data_stream/access/elasticsearch/ingest_pipeline/default.yml:160-162,224-227`

**Evidence**
- Normal flow sets `event.kind: event` (`160-162`).
- `on_failure` only sets `error.message` (`224-227`).

**What is wrong**
Failed parses are not marked as `pipeline_error`.

**Why it matters**
Nginx access logs are free-form and operationally noisy; parse failures are common under custom formats and upstream variations. Missing failure classification hides ingest parsing problems.

**Suggested fix**
In top-level `on_failure`, set `event.kind` to `pipeline_error` before/alongside `error.message`.

### 3. `apache` access pipeline `on_failure` lacks `event.kind: pipeline_error`

**Location**
- `packages/apache/data_stream/access/elasticsearch/ingest_pipeline/default.yml:95-96,248-251`

**Evidence**
- Normal flow sets `event.kind: event` (`95-96`).
- `on_failure` only sets `error.message` (`248-251`).

**What is wrong**
Failed parses are not marked as `pipeline_error`.

**Why it matters**
Apache access logs are free-form and frequently customized; parse failures are expected in real deployments. Missing failure classification reduces observability and delays troubleshooting.

**Suggested fix**
In top-level `on_failure`, set `event.kind` to `pipeline_error` before/alongside `error.message`.

## Suggested actions

- [ ] Update `system` syslog `default.yml`, `log.yml`, and `journald.yml` `on_failure` handlers to set `event.kind: pipeline_error`.
- [ ] Update `nginx` access `on_failure` to set `event.kind: pipeline_error`.
- [ ] Update `apache` access `on_failure` to set `event.kind: pipeline_error`.
- [ ] Add/adjust pipeline tests to assert `event.kind=pipeline_error` is present on parse failures for these data streams.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Pipeline Error Handler Compliance](https://github.com/elastic/integrations/actions/runs/26879064723)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 10, 2026, 10:44 AM UTC