elastic / elastic/integrations
Ingest-pipeline diagnosability fixes required for the 19 beta→GA promotions
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Context
#20418 tracks promoting 19 beta integrations to GA (1.0.0). All 19 have already flipped `deployment_modes.agentless.release: ga` while their package versions are still 0.x, so agentless-GA is in place ahead of the 1.0.0 bump. Analysis of 90-day ingest-pipeline telemetry surfaced pipeline-quality issues on several of them that should be resolved as part of that promotion.
In Agentless, when a pipeline fails, the indexed `pipeline_error` document is effectively the only forensic artifact we have. The items below make that artifact usable, and fix two confirmed defects.
### 1. Add `tag` to every processor (3 packages)
Pipeline-failure telemetry keys on the processor identifier, which is `type:tag` when a processor has a `tag` and a bare `type` otherwise. When multiple same-type processors in a pipeline are untagged, their failure counts collapse into a single bucket (e.g. all `grok` failures report as `grok`), so we cannot tell which processor is failing. The other 16 integrations in #20418 are already 89–100% tagged; three are not:
- [ ] `swimlane` — 8% of processors tagged
- [ ] `bitsight` — 30%
- [ ] `entro` — 29% (no field telemetry yet, so this is latent, but should be fixed before GA)
Action: every processor that can fail should carry a unique, descriptive `tag`.
### 2. Preserve `event.original` on pipeline error (audit all 19)
A pipeline failure should leave `event.original` on the indexed error document so the raw payload can be inspected. The standard `on_failure` block should set `event.kind: pipeline_error`, append `preserve_original_event` to `tags`, and append the processor context to `error.message` — and `event.original` must be populated and not removed before any processor that can fail.
`bitsight` is a confirmed anti-pattern: its pipeline renames `message` to `event.original`, then removes `event.original` unless the `preserve_original_event` tag is already present — but that tag is only appended later, inside `on_failure`. So for any failure after the removal step, the original payload is already gone. `swimlane`, by contrast, sets `event.original` up front and never removes it.
Action: audit all 19 to confirm `event.original` survives a mid-pipeline failure; fix ordering where it does not (starting with `bitsight`).
### Scope
The 19 integrations tracked in #20418. Broader owned-package rollout of items 1 and 2: https://github.com/elastic/integrations/issues/20558.
Contributor guide
Research direction
Start with the bitsight ingest pipeline, then compare its failure handling with swimlane and the other integrations tracked in #20418. Audit all 19 pipelines for unique processor tags and confirm that event.original survives failures; done means the required on_failure context is preserved and the identified ordering defect is fixed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, handlebars
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100