elastic / elastic/integrations
[field-mapping-conflicts] Guaranteed `event.kind` constant_keyword conflicts and int32 byte-counter overflows
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Findings
### 1) Guaranteed document rejection on parse failures: `event.kind` is `constant_keyword` but failure handlers set `pipeline_error`
These data streams declare `event.kind` as a fixed `constant_keyword`, then set `event.kind: pipeline_error` in `on_failure`. When a processor fails, Elasticsearch rejects the document because `constant_keyword` only accepts the declared constant.
#### Affected data streams (verified)
- `packages/blacklens/data_stream/alerts/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/blacklens/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml` L98-L101 (`on_failure` sets `pipeline_error`)
- `packages/github/data_stream/code_scanning/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml` L286-L289 (`on_failure` sets `pipeline_error`)
- `packages/github/data_stream/dependabot/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/dependabot/elasticsearch/ingest_pipeline/default.yml` L317-L320 (`on_failure` sets `pipeline_error`)
- `packages/github/data_stream/issues/fields/ecs.yml` L2-L4 (`value: event`)
- `packages/github/data_stream/issues/elasticsearch/ingest_pipeline/default.yml` L249-L252 (`on_failure` sets `pipeline_error`)
- `packages/github/data_stream/secret_scanning/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/secret_scanning/elasticsearch/ingest_pipeline/default.yml` L299-L302 (`on_failure` sets `pipeline_error`)
- `packages/microsoft_sentinel/data_stream/alert/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/alert/elasticsearch/ingest_pipeline/default.yml` L474-L477 (top-level `on_failure` sets `pipeline_error`)
- `packages/microsoft_sentinel/data_stream/event/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/event/elasticsearch/ingest_pipeline/default.yml` L468-L471 (top-level `on_failure` sets `pipeline_error`)
- `packages/microsoft_sentinel/data_stream/incident/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/incident/elasticsearch/ingest_pipeline/default.yml` L472-L475 (top-level `on_failure` sets `pipeline_error`)
#### Why this is high severity
`on_failure` is supposed to preserve failed documents with error context. Here it deterministically writes a value forbidden by mapping, so failed documents are rejected at index time (silent data loss from user perspective).
#### Regression notes
Recent history checked for representative affected mappings (blacklens/github/microsoft_sentinel) did not show an explicit `long->integer`-style regression marker for this conflict.
#### Suggested fix
Use one of these consistent patterns per data stream:
- Keep `constant_keyword` and stop mutating `event.kind` in failure handlers (write failure state to `event.type`, `error.*`, or `tags` instead), or
- Change `event.kind` mapping from `constant_keyword` to `keyword` where `pipeline_error` is required.
---
### 2) High-confidence integer overflow: byte totals mapped as `integer` in Arista NGFW system stats
Two byte-total fields are declared `integer` (int32), while pipeline assigns raw total disk/memory bytes. Real appliance totals commonly exceed 2,147,483,647 bytes, causing index-time numeric overflow failures.
#### Affected fields
- `arista.disk.total.bytes`
- Mapping: `packages/arista_ngfw/data_stream/log/fields/fields.yml` L56-L58 (`type: integer`, description: total disk space)
- Assignment: `packages/arista_ngfw/data_stream/log/elasticsearch/ingest_pipeline/system_stats.yml` L15-L18 (`arista.diskTotal` -> `arista.disk.total.bytes`)
- `arista.memory.total.bytes`
- Mapping: `packages/arista_ngfw/data_stream/log/fields/fields.yml` L137-L139 (`type: integer`, description: total memory in bytes)
- Assignment: `packages/arista_ngfw/data_stream/log/elasticsearch/ingest_pipeline/system_stats.yml` L80-L83 (`arista.memTotal` -> `arista.memory.total.bytes`)
#### Why this is high severity
These are total byte counters, not small bounded metrics. Production disk/RAM totals >2GB are routine, so int32 overflow is expected in normal deployments and will reject documents.
#### Regression notes
History for `packages/arista_ngfw/data_stream/log/fields/fields.yml` shows introduction in initial release; no `long -> integer` downgrade found.
#### Suggested fix
Change both fields from `integer` to `long` (or `unsigned_long` if appropriate across package conventions) and verify downstream dashboards/queries continue to work.
## Suggested Actions
- [ ] Resolve all listed `event.kind` constant conflict pairs by making mapping and failure-path values consistent.
- [ ] Update `arista.disk.total.bytes` and `arista.memory.total.bytes` mappings to a 64-bit numeric type.
- [ ] Add/extend package tests with malformed input and large byte values to ensure failures are indexed with error context, not dropped.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Field Mapping Type Conflicts](https://github.com/elastic/integrations/actions/runs/23481949153)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Mar 31, 2026, 9:29 AM UTC
Contributor guide
Research direction
Start with the listed fields.yml and default.yml files for the blacklens, github, and microsoft_sentinel data streams, then inspect arista_ngfw/data_stream/log/fields/fields.yml alongside system_stats.yml. Run the relevant package tests with malformed input and large byte values. Done means failed documents retain error context without mapping rejection and both byte totals accept production-sized values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100