elastic / elastic/elastic-package
validateIgnoredFields silently passes: runtime script never reads `_ignored`
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
## Summary
System tests are supposed to fail when indexed documents contain values in the `_ignored` metadata field, but they don't. The Painless runtime script that feeds the `ignored_fields` terms aggregation reads `_ignored` through the *stored-fields* lookup (`params['_fields']._ignored.values`), which returns nothing on at least Elasticsearch 9.3.4. The terms aggregation produces zero buckets, `validateIgnoredFields` sees an empty list, and the test passes — even when every document in the data stream has `_ignored` populated.
This defeats the entire `_ignored`-fields safety net that was added in #1738 ("Extend system test to validate absence of `_ignored`", b33490b6).
## How it was detected
While triaging an unrelated bug in the `netflow` integration, `elastic-package test system --data-streams log` against `packages/netflow` v2.25.0 **passed** on a 9.3.4 stack despite the indexed documents containing:
```json
{
"_index": ".ds-logs-netflow.log-79901-2026.05.05-000001",
"_ignored": ["event.created", "netflow.exporter.timestamp"],
"_source": {
"event": { "created": {} },
"netflow": { "exporter": { "timestamp": {} } }
},
"ignored_field_values": {
"event.created": [{}],
"netflow.exporter.timestamp": [{}]
}
}
```
All 29 documents in the data stream had `_ignored` set on both fields. The package does **not** configure `skip_ignored_fields`, both fields are declared as `date` (`event.created` via ECS import, `netflow.exporter.timestamp` in `package-fields.yml`), and the test config does not set `skip_reason`/`skip_link`. The validation should have fired but didn't.
## Why it's failing
The `_ignored` metadata field is exposed via doc-values, **not** stored fields. On Elasticsearch 9.3.4, `params['_fields']['_ignored']` resolves to a `FieldLookup` whose `getValues()` returns an empty list for `_ignored`, even on documents whose search response clearly includes `"_ignored": [...]`.
Reproduction against the same cluster, same index, same query body as elastic-package:
```bash
# 1. Confirm 29 docs have _ignored populated
$ curl -sk -u elastic:changeme \
'https://localhost:9200/logs-netflow.log-*/_search?size=0' \
-H 'Content-Type: application/json' \
-d '{"query":{"exists":{"field":"_ignored"}}}'
# → "hits": { "total": { "value": 29, "relation": "eq" } }
# 2. Run the exact aggregation from internal/testrunner/runners/system/tester.go
$ curl -sk -u elastic:changeme \
'https://localhost:9200/logs-netflow.log-*/_search?size=0' \
-H 'Content-Type: application/json' \
-d '{
"runtime_mappings": {
"my_ignored": {
"type": "keyword",
"script": { "source": "for (def v : params[\"_fields\"]._ignored.values) { emit(v); }" }
}
},
"aggs": {
"all_ignored": {
"filter": { "exists": { "field": "_ignored" } },
"aggs": { "ignored_fields": { "terms": { "size": 100, "field": "my_ignored" } } }
}
}
}'
# → all_ignored.doc_count = 29
# → all_ignored.ignored_fields.buckets = [] ← BUG: no field names emitted
# 3. Same aggregation with doc['_ignored'] instead
# "source": "for (def v : doc['_ignored']) { emit(v); }"
# → buckets: [
# { "key": "event.created", "doc_count": 29 },
# { "key": "netflow.exporter.timestamp", "doc_count": 29 }
# ]
```
Direct introspection confirms `params['_fields']['_ignored'].getValues().size() == 0` on documents whose response includes `_ignored: ["event.created", "netflow.exporter.timestamp"]`.
### Resulting code path
1. `getDocs()` runs the search — 29 hits, `all_ignored` filter agg counts 29, **but the inner `ignored_fields` terms agg returns zero buckets** because the `my_ignored` runtime field never emits.
2. `tester.go:880-881` writes nothing into `hits.IgnoredFields`.
3. `tester.go:1207` sets `sds.ignoredFields = []`.
4. `validateIgnoredFields` (called unconditionally at `tester.go:1990`) sees `len(ds.ignoredFields) == 0` and returns `nil`.
5. The test passes.
## Affected source
- `FieldsQuery` constant containing the broken Painless script:
- The single offending line:
## Proposed fix
Switch the runtime field to read `_ignored` from doc-values:
```diff
--- a/internal/testrunner/runners/system/tester.go
+++ b/internal/testrunner/runners/system/tester.go
@@ -50,7 +50,7 @@ const FieldsQuery = `{
"my_ignored": {
"type": "keyword",
"script": {
- "source": "for (def v : params['_fields']._ignored.values) { emit(v); }"
+ "source": "for (def v : doc['_ignored']) { emit(v); }"
}
}
},
```
`doc['_ignored']` returns the field names correctly; the rest of the aggregation, `getDocs`, and `validateIgnoredFields` are unchanged. Verified against Elasticsearch 9.3.4 (build hash `69a3e6c50ebb57a1fdbf3f235be9f11061ac7d86`).
## Suggested follow-up
- Add a regression test (or integration test) that asserts a system test fails when the indexed doc has `_ignored` populated. The current unit tests cover `validateIgnoredFields`'s post-aggregation behaviour but not the search query, so this regression slipped past CI.
- Audit how widely the bug has masked failures by re-running system tests for packages that rely on `ignore_malformed: true` from `logs@settings` (most logs integrations). It is likely that other packages have been silently shipping schema drift.
## Impact
Every system test relying on `_ignored` validation has been a no-op on stacks where `_ignored` is not exposed via the stored-fields lookup. For data streams using the `logs@settings` defaults (which set `ignore_malformed: true` on date/numeric fields), this means real schema mismatches — a `date` field receiving an object, a `long` receiving a string — are silently absorbed by the index instead of surfacing as test failures.
## Environment
- elastic-package: `main` @ `5b266c6` (also affects all releases since #1738)
- Elasticsearch: 9.3.4 (build hash `69a3e6c50ebb57a1fdbf3f235be9f11061ac7d86`)
- Reproducing package: `elastic/integrations` `packages/netflow` v2.25.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in internal/testrunner/runners/system/tester.go, especially the FieldsQuery constant and the getDocs and validateIgnoredFields paths named in the issue. Run the provided Elasticsearch aggregation reproduction or the existing system-runner tests, then verify that documents with populated _ignored produce field buckets and cause validation to fail when appropriate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, go
- Domain
- cli, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- Half a day
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100