elastic / elastic/integrations

SSI: Standardize processor tagging and event.original preservation across owned pipelines

Closed
#20,558 2 comments 0 reactions 1 assignee Claimed by @kcreddy View on GitHub
enhancement Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Context

Diagnosing ingest-pipeline failures from telemetry depends on two conventions being applied consistently: every processor carrying a `tag`, and `event.original` being preserved on failure. A scan of the 191 packages owned by `elastic/security-service-integrations` shows both are applied unevenly. This issue tracks bringing the whole set up to standard; it generalizes the GA-candidate work in https://github.com/elastic/integrations/issues/20557.

### 1. Processor tagging

Failure telemetry identifies a processor as `type:tag`, or a bare `type` when untagged. Untagged same-type processors collapse into one bucket, so failures cannot be attributed to a specific step.

Current state across 191 packages: **69% of processors tagged** in aggregate.

- **6 packages with zero tagged processors:** `carbonblack_edr`, `cylance`, `jamf_compliance_reporter`, `jamf_protect`, `lumos`, `lyve_cloud`
- **80 packages below 50%.** Worst offenders (≤5%): `infoblox_bloxone_ddi` (1%), `sophos_central` (1%), `atlassian_confluence` (2%), `carbon_black_cloud` (2%), `darktrace` (2%), `mattermost` (2%), `santa` (2%), `tenable_sc` (2%), `thycotic_ss` (2%), `ti_eclecticiq` (2%), `ti_mandiant_advantage` (2%), `1password` (3%), `bbot` (3%), `lastpass` (3%), `proofpoint_tap` (3%), `zerofox` (3%), `zscaler_zpa` (3%), … (full list in the scan output).

Target: every failure-capable processor carries a unique descriptive `tag`.

### 2. event.original preservation on failure

Of 797 pipelines with an `on_failure` block, roughly a quarter do not reference `preserve_original_event`, meaning a failure may not retain the raw payload for diagnosis. Ordering bugs also exist where `event.original` is removed before the failure point (see `bitsight` in ).

Target: every pipeline's `on_failure` sets `event.kind: pipeline_error`, appends `preserve_original_event` to `tags`, and appends processor context to `error.message`; and `event.original` is populated before, and not removed ahead of, any processor that can fail.

### Suggested approach

- Prioritize the agentless GA candidates (https://github.com/elastic/integrations/issues/20557) and the zero-tag / sub-5% packages first.
- Automation exists that can add `event.original` preservation; worth reusing here.
- `elastic-package` can add tags using: `elastic-package modify -m pipeline-tag`
- Consider a lint/CI check so new pipelines can't regress (untagged processors, missing `preserve_original_event` in `on_failure`).

### Measurement

Numbers above are from parsing top-level processors in `packages/*/data_stream/*/elasticsearch/ingest_pipeline/*.yml` on `main`. Percentages exclude processors nested inside `foreach`/`on_failure`, and the `event.original` figure is a heuristic — treat per-package status as a starting point to verify, not a certification.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.