elastic / elastic/elastic-package

[system tests] Ignored fields are not detected in stacks 8.15.0 onward

Open
#2,434 2 comments 0 reactions 0 assignees View on GitHub
Team:Ecosystem
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

The current search performed to get the ignored fields in a given data stream is not working successfully for Elastic stack versions: 8.15.0

https://github.com/elastic/elastic-package/blob/091c6636bb00aa2acbbb2c2120c76c65eb6392e3/internal/testrunner/runners/system/tester.go#L46-L78

For Elastic stack >= 8.15.0, that `my_ignored` runtime field is not populated and it is not present in the search response, even for the data streams that contain some ignored field.

Example: this build from windows package (integrations repo) fails with 8.14.0, but it finishes successfully when it runs with 8.15.0 or 8.17.0
https://buildkite.com/elastic/integrations/builds/22752

To be tested if this query works for >=8.15.0

```json
{
"fields": ["*"],
"aggs": {
"all_ignored": {
"filter": {
"exists": {
"field": "_ignored"
}
},
"aggs": {
"ignored_fields": {
"terms": {
"size": 100,
"field": "_ignored"
}
},
"ignored_docs": {
"top_hits": {
"size": 5
}
}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/testrunner/runners/system/tester.go at lines 46-78 and reproduce the ignored-field search against Elastic stack versions 8.15.0 and later. Test the proposed aggregation query and compare its response with the current search, using the Windows package build as a reference. Done means system tests detect ignored fields consistently on the affected versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.