elastic / elastic/integrations

Ingest-pipeline diagnosability fixes required for the 19 beta→GA promotions

Open
#20,557 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Context

#20418 tracks promoting 19 beta integrations to GA (1.0.0). All 19 have already flipped `deployment_modes.agentless.release: ga` while their package versions are still 0.x, so agentless-GA is in place ahead of the 1.0.0 bump. Analysis of 90-day ingest-pipeline telemetry surfaced pipeline-quality issues on several of them that should be resolved as part of that promotion.

In Agentless, when a pipeline fails, the indexed `pipeline_error` document is effectively the only forensic artifact we have. The items below make that artifact usable, and fix two confirmed defects.

### 1. Add `tag` to every processor (3 packages)

Pipeline-failure telemetry keys on the processor identifier, which is `type:tag` when a processor has a `tag` and a bare `type` otherwise. When multiple same-type processors in a pipeline are untagged, their failure counts collapse into a single bucket (e.g. all `grok` failures report as `grok`), so we cannot tell which processor is failing. The other 16 integrations in #20418 are already 89–100% tagged; three are not:

- [ ] `swimlane` — 8% of processors tagged
- [ ] `bitsight` — 30%
- [ ] `entro` — 29% (no field telemetry yet, so this is latent, but should be fixed before GA)

Action: every processor that can fail should carry a unique, descriptive `tag`.

### 2. Preserve `event.original` on pipeline error (audit all 19)

A pipeline failure should leave `event.original` on the indexed error document so the raw payload can be inspected. The standard `on_failure` block should set `event.kind: pipeline_error`, append `preserve_original_event` to `tags`, and append the processor context to `error.message` — and `event.original` must be populated and not removed before any processor that can fail.

`bitsight` is a confirmed anti-pattern: its pipeline renames `message` to `event.original`, then removes `event.original` unless the `preserve_original_event` tag is already present — but that tag is only appended later, inside `on_failure`. So for any failure after the removal step, the original payload is already gone. `swimlane`, by contrast, sets `event.original` up front and never removes it.

Action: audit all 19 to confirm `event.original` survives a mid-pipeline failure; fix ordering where it does not (starting with `bitsight`).

### Scope

The 19 integrations tracked in #20418. Broader owned-package rollout of items 1 and 2: https://github.com/elastic/integrations/issues/20558.

Contributor guide

Open the contributing guide

Research direction

Start with the bitsight ingest pipeline, then compare its failure handling with swimlane and the other integrations tracked in #20418. Audit all 19 pipelines for unique processor tags and confirm that event.original survives failures; done means the required on_failure context is preserved and the identified ordering defect is fixed.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, handlebars
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.