elastic / elastic/integrations
SSI: Standardize processor tagging and event.original preservation across owned pipelines
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Context
Diagnosing ingest-pipeline failures from telemetry depends on two conventions being applied consistently: every processor carrying a `tag`, and `event.original` being preserved on failure. A scan of the 191 packages owned by `elastic/security-service-integrations` shows both are applied unevenly. This issue tracks bringing the whole set up to standard; it generalizes the GA-candidate work in https://github.com/elastic/integrations/issues/20557.
### 1. Processor tagging
Failure telemetry identifies a processor as `type:tag`, or a bare `type` when untagged. Untagged same-type processors collapse into one bucket, so failures cannot be attributed to a specific step.
Current state across 191 packages: **69% of processors tagged** in aggregate.
- **6 packages with zero tagged processors:** `carbonblack_edr`, `cylance`, `jamf_compliance_reporter`, `jamf_protect`, `lumos`, `lyve_cloud`
- **80 packages below 50%.** Worst offenders (≤5%): `infoblox_bloxone_ddi` (1%), `sophos_central` (1%), `atlassian_confluence` (2%), `carbon_black_cloud` (2%), `darktrace` (2%), `mattermost` (2%), `santa` (2%), `tenable_sc` (2%), `thycotic_ss` (2%), `ti_eclecticiq` (2%), `ti_mandiant_advantage` (2%), `1password` (3%), `bbot` (3%), `lastpass` (3%), `proofpoint_tap` (3%), `zerofox` (3%), `zscaler_zpa` (3%), … (full list in the scan output).
Target: every failure-capable processor carries a unique descriptive `tag`.
### 2. event.original preservation on failure
Of 797 pipelines with an `on_failure` block, roughly a quarter do not reference `preserve_original_event`, meaning a failure may not retain the raw payload for diagnosis. Ordering bugs also exist where `event.original` is removed before the failure point (see `bitsight` in ).
Target: every pipeline's `on_failure` sets `event.kind: pipeline_error`, appends `preserve_original_event` to `tags`, and appends processor context to `error.message`; and `event.original` is populated before, and not removed ahead of, any processor that can fail.
### Suggested approach
- Prioritize the agentless GA candidates (https://github.com/elastic/integrations/issues/20557) and the zero-tag / sub-5% packages first.
- Automation exists that can add `event.original` preservation; worth reusing here.
- `elastic-package` can add tags using: `elastic-package modify -m pipeline-tag`
- Consider a lint/CI check so new pipelines can't regress (untagged processors, missing `preserve_original_event` in `on_failure`).
### Measurement
Numbers above are from parsing top-level processors in `packages/*/data_stream/*/elasticsearch/ingest_pipeline/*.yml` on `main`. Percentages exclude processors nested inside `foreach`/`on_failure`, and the `event.original` figure is a heuristic — treat per-package status as a starting point to verify, not a certification.
Contributor guide
Assessment
This issue has not been assessed yet.