elastic / elastic/elastic-package

Source Index Queries in Transform not Applied During System Tests

Open
#2,676 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

This issue outlines a discrepancy between the behavior of a transform using a **source index** with **query** conditions in actual usage versus its behavior during system tests.

We have configured a transform that uses a source index with a query to filter specific types of documents. The purpose is to ensure only relevant documents are processed by the transform.

```yml
source:
index:
- "logs-ti_google_threat_intelligence.*-*"
query:
bool:
filter:
- match_phrase:
threat.indicator.type: domain-name
must_not:
- exists:
field: error.message
- match_phrase:
event.dataset: ti_google_threat_intelligence.ioc_stream
- match_phrase:
event.dataset: ti_google_threat_intelligence.enriched_ioc
- match_phrase:
event.dataset: ti_google_threat_intelligence.enriched_ioc_stream
```

When the transform is executed in a live Elastic instance:
- The query is applied correctly.
- Only appropriate documents are used in transforms.
- The filtered documents are indexed into the appropriate destination index.
Manual verification confirms that indexing is working as intended.

![Image](https://github.com/user-attachments/assets/99b180f1-66d2-4056-80ce-b0373f438a28)

However, when we test the implementation using the system test framework, inappropriate documents are sourced to the transform. It appears that the query defined in the transform is not being applied correctly, leading to incorrect indexing and system test failures.

```
--- Test results for package: ti_google_threat_intelligence - START ---
╭───────────────────────────────┬─────────────┬───────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├───────────────────────────────┼─────────────┼───────────┼───────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────┤
│ ti_google_threat_intelligence │ iot │ system │ default │ ERROR: no documents found in preview for transform "logs-ti_google_threat_intelligence.domain_ioc_st-default-0.1.0" │ 2m10.257972784s │
╰───────────────────────────────┴─────────────┴───────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────╯
--- Test results for package: ti_google_threat_intelligence - END ---
Done
Error: failed to process results: one or more test cases failed
```

Relates.
https://github.com/elastic/integrations/pull/13189

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the system test framework's transform preview path and reproduce the reported source-index query case for the ti_google_threat_intelligence package. Compare the preview behavior with the live transform, then verify that the source query filters documents and the system test no longer reports an empty preview.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.