elastic / elastic/elastic-package
Source Index Queries in Transform not Applied During System Tests
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
This issue outlines a discrepancy between the behavior of a transform using a **source index** with **query** conditions in actual usage versus its behavior during system tests.
We have configured a transform that uses a source index with a query to filter specific types of documents. The purpose is to ensure only relevant documents are processed by the transform.
```yml
source:
index:
- "logs-ti_google_threat_intelligence.*-*"
query:
bool:
filter:
- match_phrase:
threat.indicator.type: domain-name
must_not:
- exists:
field: error.message
- match_phrase:
event.dataset: ti_google_threat_intelligence.ioc_stream
- match_phrase:
event.dataset: ti_google_threat_intelligence.enriched_ioc
- match_phrase:
event.dataset: ti_google_threat_intelligence.enriched_ioc_stream
```
When the transform is executed in a live Elastic instance:
- The query is applied correctly.
- Only appropriate documents are used in transforms.
- The filtered documents are indexed into the appropriate destination index.
Manual verification confirms that indexing is working as intended.

However, when we test the implementation using the system test framework, inappropriate documents are sourced to the transform. It appears that the query defined in the transform is not being applied correctly, leading to incorrect indexing and system test failures.
```
--- Test results for package: ti_google_threat_intelligence - START ---
╭───────────────────────────────┬─────────────┬───────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├───────────────────────────────┼─────────────┼───────────┼───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
│ ti_google_threat_intelligence │ iot │ system │ default │ ERROR: no documents found in preview for transform "logs-ti_google_threat_intelligence.domain_ioc_st-default-0.1.0" │ 2m10.257972784s │
╰───────────────────────────────┴─────────────┴───────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────╯
--- Test results for package: ti_google_threat_intelligence - END ---
Done
Error: failed to process results: one or more test cases failed
```
Relates.
https://github.com/elastic/integrations/pull/13189
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the system test framework's transform preview path and reproduce the reported source-index query case for the ti_google_threat_intelligence package. Compare the preview behavior with the live transform, then verify that the source query filters documents and the system test no longer reports an empty preview.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100