elastic / elastic/integrations

[field-mapping-conflicts] Guaranteed `event.kind` constant_keyword conflicts and int32 byte-counter overflows

Open
#18,001 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

## Findings

### 1) Guaranteed document rejection on parse failures: `event.kind` is `constant_keyword` but failure handlers set `pipeline_error`

These data streams declare `event.kind` as a fixed `constant_keyword`, then set `event.kind: pipeline_error` in `on_failure`. When a processor fails, Elasticsearch rejects the document because `constant_keyword` only accepts the declared constant.

#### Affected data streams (verified)

- `packages/blacklens/data_stream/alerts/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/blacklens/data_stream/alerts/elasticsearch/ingest_pipeline/default.yml` L98-L101 (`on_failure` sets `pipeline_error`)

- `packages/github/data_stream/code_scanning/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/code_scanning/elasticsearch/ingest_pipeline/default.yml` L286-L289 (`on_failure` sets `pipeline_error`)

- `packages/github/data_stream/dependabot/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/dependabot/elasticsearch/ingest_pipeline/default.yml` L317-L320 (`on_failure` sets `pipeline_error`)

- `packages/github/data_stream/issues/fields/ecs.yml` L2-L4 (`value: event`)
- `packages/github/data_stream/issues/elasticsearch/ingest_pipeline/default.yml` L249-L252 (`on_failure` sets `pipeline_error`)

- `packages/github/data_stream/secret_scanning/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/github/data_stream/secret_scanning/elasticsearch/ingest_pipeline/default.yml` L299-L302 (`on_failure` sets `pipeline_error`)

- `packages/microsoft_sentinel/data_stream/alert/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/alert/elasticsearch/ingest_pipeline/default.yml` L474-L477 (top-level `on_failure` sets `pipeline_error`)

- `packages/microsoft_sentinel/data_stream/event/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/event/elasticsearch/ingest_pipeline/default.yml` L468-L471 (top-level `on_failure` sets `pipeline_error`)

- `packages/microsoft_sentinel/data_stream/incident/fields/ecs.yml` L2-L4 (`value: alert`)
- `packages/microsoft_sentinel/data_stream/incident/elasticsearch/ingest_pipeline/default.yml` L472-L475 (top-level `on_failure` sets `pipeline_error`)

#### Why this is high severity

`on_failure` is supposed to preserve failed documents with error context. Here it deterministically writes a value forbidden by mapping, so failed documents are rejected at index time (silent data loss from user perspective).

#### Regression notes

Recent history checked for representative affected mappings (blacklens/github/microsoft_sentinel) did not show an explicit `long->integer`-style regression marker for this conflict.

#### Suggested fix

Use one of these consistent patterns per data stream:

- Keep `constant_keyword` and stop mutating `event.kind` in failure handlers (write failure state to `event.type`, `error.*`, or `tags` instead), or
- Change `event.kind` mapping from `constant_keyword` to `keyword` where `pipeline_error` is required.

---

### 2) High-confidence integer overflow: byte totals mapped as `integer` in Arista NGFW system stats

Two byte-total fields are declared `integer` (int32), while pipeline assigns raw total disk/memory bytes. Real appliance totals commonly exceed 2,147,483,647 bytes, causing index-time numeric overflow failures.

#### Affected fields

- `arista.disk.total.bytes`
- Mapping: `packages/arista_ngfw/data_stream/log/fields/fields.yml` L56-L58 (`type: integer`, description: total disk space)
- Assignment: `packages/arista_ngfw/data_stream/log/elasticsearch/ingest_pipeline/system_stats.yml` L15-L18 (`arista.diskTotal` -> `arista.disk.total.bytes`)

- `arista.memory.total.bytes`
- Mapping: `packages/arista_ngfw/data_stream/log/fields/fields.yml` L137-L139 (`type: integer`, description: total memory in bytes)
- Assignment: `packages/arista_ngfw/data_stream/log/elasticsearch/ingest_pipeline/system_stats.yml` L80-L83 (`arista.memTotal` -> `arista.memory.total.bytes`)

#### Why this is high severity

These are total byte counters, not small bounded metrics. Production disk/RAM totals >2GB are routine, so int32 overflow is expected in normal deployments and will reject documents.

#### Regression notes

History for `packages/arista_ngfw/data_stream/log/fields/fields.yml` shows introduction in initial release; no `long -> integer` downgrade found.

#### Suggested fix

Change both fields from `integer` to `long` (or `unsigned_long` if appropriate across package conventions) and verify downstream dashboards/queries continue to work.

## Suggested Actions

- [ ] Resolve all listed `event.kind` constant conflict pairs by making mapping and failure-path values consistent.
- [ ] Update `arista.disk.total.bytes` and `arista.memory.total.bytes` mappings to a 64-bit numeric type.
- [ ] Add/extend package tests with malformed input and large byte values to ensure failures are indexed with error context, not dropped.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Field Mapping Type Conflicts](https://github.com/elastic/integrations/actions/runs/23481949153)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Mar 31, 2026, 9:29 AM UTC

Contributor guide

Open the contributing guide

Research direction

Start with the listed fields.yml and default.yml files for the blacklens, github, and microsoft_sentinel data streams, then inspect arista_ngfw/data_stream/log/fields/fields.yml alongside system_stats.yml. Run the relevant package tests with malformed input and large byte values. Done means failed documents retain error context without mapping rejection and both byte totals accept production-sized values.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.