elastic / elastic/elastic-package

validateIgnoredFields silently passes: runtime script never reads `_ignored`

Open
#3,524 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

## Summary

System tests are supposed to fail when indexed documents contain values in the `_ignored` metadata field, but they don't. The Painless runtime script that feeds the `ignored_fields` terms aggregation reads `_ignored` through the *stored-fields* lookup (`params['_fields']._ignored.values`), which returns nothing on at least Elasticsearch 9.3.4. The terms aggregation produces zero buckets, `validateIgnoredFields` sees an empty list, and the test passes — even when every document in the data stream has `_ignored` populated.

This defeats the entire `_ignored`-fields safety net that was added in #1738 ("Extend system test to validate absence of `_ignored`", b33490b6).

## How it was detected

While triaging an unrelated bug in the `netflow` integration, `elastic-package test system --data-streams log` against `packages/netflow` v2.25.0 **passed** on a 9.3.4 stack despite the indexed documents containing:

```json
{
"_index": ".ds-logs-netflow.log-79901-2026.05.05-000001",
"_ignored": ["event.created", "netflow.exporter.timestamp"],
"_source": {
"event": { "created": {} },
"netflow": { "exporter": { "timestamp": {} } }
},
"ignored_field_values": {
"event.created": [{}],
"netflow.exporter.timestamp": [{}]
}
}
```

All 29 documents in the data stream had `_ignored` set on both fields. The package does **not** configure `skip_ignored_fields`, both fields are declared as `date` (`event.created` via ECS import, `netflow.exporter.timestamp` in `package-fields.yml`), and the test config does not set `skip_reason`/`skip_link`. The validation should have fired but didn't.

## Why it's failing

The `_ignored` metadata field is exposed via doc-values, **not** stored fields. On Elasticsearch 9.3.4, `params['_fields']['_ignored']` resolves to a `FieldLookup` whose `getValues()` returns an empty list for `_ignored`, even on documents whose search response clearly includes `"_ignored": [...]`.

Reproduction against the same cluster, same index, same query body as elastic-package:

```bash
# 1. Confirm 29 docs have _ignored populated
$ curl -sk -u elastic:changeme \
'https://localhost:9200/logs-netflow.log-*/_search?size=0' \
-H 'Content-Type: application/json' \
-d '{"query":{"exists":{"field":"_ignored"}}}'
# → "hits": { "total": { "value": 29, "relation": "eq" } }

# 2. Run the exact aggregation from internal/testrunner/runners/system/tester.go
$ curl -sk -u elastic:changeme \
'https://localhost:9200/logs-netflow.log-*/_search?size=0' \
-H 'Content-Type: application/json' \
-d '{
"runtime_mappings": {
"my_ignored": {
"type": "keyword",
"script": { "source": "for (def v : params[\"_fields\"]._ignored.values) { emit(v); }" }
}
},
"aggs": {
"all_ignored": {
"filter": { "exists": { "field": "_ignored" } },
"aggs": { "ignored_fields": { "terms": { "size": 100, "field": "my_ignored" } } }
}
}
}'
# → all_ignored.doc_count = 29
# → all_ignored.ignored_fields.buckets = [] ← BUG: no field names emitted

# 3. Same aggregation with doc['_ignored'] instead
# "source": "for (def v : doc['_ignored']) { emit(v); }"
# → buckets: [
# { "key": "event.created", "doc_count": 29 },
# { "key": "netflow.exporter.timestamp", "doc_count": 29 }
# ]
```

Direct introspection confirms `params['_fields']['_ignored'].getValues().size() == 0` on documents whose response includes `_ignored: ["event.created", "netflow.exporter.timestamp"]`.

### Resulting code path

1. `getDocs()` runs the search — 29 hits, `all_ignored` filter agg counts 29, **but the inner `ignored_fields` terms agg returns zero buckets** because the `my_ignored` runtime field never emits.

2. `tester.go:880-881` writes nothing into `hits.IgnoredFields`.

3. `tester.go:1207` sets `sds.ignoredFields = []`.

4. `validateIgnoredFields` (called unconditionally at `tester.go:1990`) sees `len(ds.ignoredFields) == 0` and returns `nil`.

5. The test passes.

## Affected source

- `FieldsQuery` constant containing the broken Painless script:

- The single offending line:

## Proposed fix

Switch the runtime field to read `_ignored` from doc-values:

```diff
--- a/internal/testrunner/runners/system/tester.go
+++ b/internal/testrunner/runners/system/tester.go
@@ -50,7 +50,7 @@ const FieldsQuery = `{
"my_ignored": {
"type": "keyword",
"script": {
- "source": "for (def v : params['_fields']._ignored.values) { emit(v); }"
+ "source": "for (def v : doc['_ignored']) { emit(v); }"
}
}
},
```

`doc['_ignored']` returns the field names correctly; the rest of the aggregation, `getDocs`, and `validateIgnoredFields` are unchanged. Verified against Elasticsearch 9.3.4 (build hash `69a3e6c50ebb57a1fdbf3f235be9f11061ac7d86`).

## Suggested follow-up

- Add a regression test (or integration test) that asserts a system test fails when the indexed doc has `_ignored` populated. The current unit tests cover `validateIgnoredFields`'s post-aggregation behaviour but not the search query, so this regression slipped past CI.
- Audit how widely the bug has masked failures by re-running system tests for packages that rely on `ignore_malformed: true` from `logs@settings` (most logs integrations). It is likely that other packages have been silently shipping schema drift.

## Impact

Every system test relying on `_ignored` validation has been a no-op on stacks where `_ignored` is not exposed via the stored-fields lookup. For data streams using the `logs@settings` defaults (which set `ignore_malformed: true` on date/numeric fields), this means real schema mismatches — a `date` field receiving an object, a `long` receiving a string — are silently absorbed by the index instead of surfacing as test failures.

## Environment

- elastic-package: `main` @ `5b266c6` (also affects all releases since #1738)
- Elasticsearch: 9.3.4 (build hash `69a3e6c50ebb57a1fdbf3f235be9f11061ac7d86`)
- Reproducing package: `elastic/integrations` `packages/netflow` v2.25.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in internal/testrunner/runners/system/tester.go, especially the FieldsQuery constant and the getDocs and validateIgnoredFields paths named in the issue. Run the provided Elasticsearch aggregation reproduction or the existing system-runner tests, then verify that documents with populated _ignored produce field buckets and cause validation to fail when appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go
Domain
cli, testing
Issue type
Bug
Difficulty
2/5
Estimated time
Half a day
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.