elastic / elastic/integrations
Add script tests to verify routing rules across integration packages
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Problem
Standard system tests only check for hits in the source data stream. Packages with routing rules cannot verify that documents actually reach their target data streams. This was identified in https://github.com/elastic/elastic-package/issues/1917.
## Solution
Use `elastic-package test script` (txtar-based tests) that call `get_docs` against target data stream index patterns. The approach was proven in https://github.com/elastic/integrations/pull/18235 for the `entityanalytics_ad` package.
## Exemplar
`packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/routed_data_streams.txt`
## Scope
Add routing-rule script tests to 10 packages owned by elastic/security-service-integrations. Each package is a separate PR.
### Tier 1 — Existing Docker mocks (symlink `_dev/deploy/docker`)
| Package | Source DS | Target DS(s) | Routing condition |
|---|---|---|---|
| entityanalytics_entra_id | entity | user, device | `ctx.user?.id`, `ctx.device?.id` |
| entityanalytics_okta | entity | user, device | `ctx.user?.id`, `ctx.device?.id` |
| cyberarkpas | audit | monitor | `ctx.message?.contains('"Product":"VaultMonitor"')` |
| doppler | activity | secret_read | `ctx.event?.action == 'security.secret_read'` |
### Tier 2 — Existing system tests, non-Docker deploy (need embedded mock)
| Package | Source DS | Target DS(s) | Notes |
|---|---|---|---|
| sentinel_one_cloud_funnel | event | 13 category-based targets | Mock emitting `sentinel_one_cloud_funnel.event.category` + `rerouted: true` |
### Tier 3 — No system tests (need mock from scratch)
| Package | Source DS | Target DS(s) | Input type |
|---|---|---|---|
| amazon_security_lake | event | 6 OCSF category targets | aws-s3 |
| azure | events | 12 azure.* targets | azure-eventhub |
| jupiter_one | asset | risks_and_alerts | cel |
| netskope | alerts_events_v2 | alerts_v2, events_v2 | cel/http_endpoint |
| proofpoint_essentials | threat | 4 event_type targets | httpjson |
Contributor guide
Research direction
Start with packages/entityanalytics_ad/data_stream/entity/_dev/test/scripts/routed_data_streams.txt and the elastic-package test script documentation. Inspect each listed package's existing _dev deployment or system-test setup, then add a package-specific txtar test or mock as needed. Done means get_docs verifies documents in every listed target data stream for the routing conditions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100