elastic / elastic/integrations

Add script tests to verify routing rules across integration packages

Open
#20,449 1 comment 0 reactions 0 assignees View on GitHub
enhancement Team:Security-Service Integrations
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Problem

Standard system tests only check for hits in the source data stream. Packages with routing rules cannot verify that documents actually reach their target data streams. This was identified in https://github.com/elastic/elastic-package/issues/1917.

## Solution

Use `elastic-package test script` (txtar-based tests) that call `get_docs` against target data stream index patterns. The approach was proven in https://github.com/elastic/integrations/pull/18235 for the `entityanalytics_ad` package.

## Exemplar

`packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/routed_data_streams.txt`

## Scope

Add routing-rule script tests to 10 packages owned by elastic/security-service-integrations. Each package is a separate PR.

### Tier 1 — Existing Docker mocks (symlink `_dev/deploy/docker`)

| Package | Source DS | Target DS(s) | Routing condition |
|---|---|---|---|
| entityanalytics_entra_id | entity | user, device | `ctx.user?.id`, `ctx.device?.id` |
| entityanalytics_okta | entity | user, device | `ctx.user?.id`, `ctx.device?.id` |
| cyberarkpas | audit | monitor | `ctx.message?.contains('"Product":"VaultMonitor"')` |
| doppler | activity | secret_read | `ctx.event?.action == 'security.secret_read'` |

### Tier 2 — Existing system tests, non-Docker deploy (need embedded mock)

| Package | Source DS | Target DS(s) | Notes |
|---|---|---|---|
| sentinel_one_cloud_funnel | event | 13 category-based targets | Mock emitting `sentinel_one_cloud_funnel.event.category` + `rerouted: true` |

### Tier 3 — No system tests (need mock from scratch)

| Package | Source DS | Target DS(s) | Input type |
|---|---|---|---|
| amazon_security_lake | event | 6 OCSF category targets | aws-s3 |
| azure | events | 12 azure.* targets | azure-eventhub |
| jupiter_one | asset | risks_and_alerts | cel |
| netskope | alerts_events_v2 | alerts_v2, events_v2 | cel/http_endpoint |
| proofpoint_essentials | threat | 4 event_type targets | httpjson |

Contributor guide

Open the contributing guide

Research direction

Start with packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/routed_data_streams.txt and the elastic-package test script documentation. Inspect each listed package's existing _dev deployment or system-test setup, then add a package-specific txtar test or mock as needed. Done means get_docs verifies documents in every listed target data stream for the routing conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
data-engineering, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.