elastic / elastic/beats

Large integers from CEL arrive in ES as Doubles

Open
#43,659 13 comments 0 reactions 1 assignee Claimed by @efd6 View on GitHub
Filebeat Team:Security-Service Integrations
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Elasticsearch is okay with integers like `235549249` and larger, and can convert them to strings.

Expand for an Elasticsearch example

```
PUT _ingest/pipeline/test_convert
{
"description": "Pipeline to test conversion of large integers",
"processors": [
{
"convert": {
"field": "small",
"target_field": "small_string",
"type": "string"
}
},
{
"convert": {
"field": "large",
"target_field": "large_string",
"type": "string"
}
}
]
}

POST _ingest/pipeline/test_convert/_simulate
{
"docs": [
{
"_source": {
"small": 12345,
"large": 2355492490000000000
}
}
]
}
```

```
{
"docs": [
{
"doc": {
"_index": "_index",
"_version": "-3",
"_id": "_id",
"_source": {
"small": 12345,
"large": 2355492490000000000,
"small_string": "12345",
"large_string": "2355492490000000000"
},
"_ingest": {
"timestamp": "2025-04-03T08:58:03.968461461Z"
}
}
}
]
}
```

However, when receiving such a number from Beats (at least from the CEL input), it is of type `Double`.

The [convert processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/convert-processor.html) will produce the string `"2.35549249E8"`. Trying to convert to an integer before converting to a string fails with an error message saying it can't convert the string "2.35549249E8" (so, the convert processor doesn't natively convert from Double to integer).

Expand for a script processor that does a type check and conversion in Painless

```
- script:
description: Convert large integer to a string without an exponent
lang: painless
source: |
def value = (Object) ctx.json.id;
if (value instanceof Integer) {
ctx.json["id_type_name"] = "Integer";
} else if (value instanceof Double) {
ctx.json["id_type_name"] = "Double";
} else if (value instanceof String) {
ctx.json["id_type_name"] = "String";
} else {
ctx.json["id_type_name"] = "Unknown";
}
ctx.json.id = Long.toString((long) ctx.json.id);
```

Result:
```
json.id_type_name = "Double"
json.id = 235549249
```

This requires special handling, such as [in the `ti_anomali` integration](https://github.com/elastic/integrations/blob/703b76e5b9004bc2c9c06e5844e6a56465b0cb4c/packages/ti_anomali/data_stream/intelligence/elasticsearch/ingest_pipeline/default.yml#L524-L529). In that case, notice that the original server response is an integer in JSON (e.g. `"id":235548914` in the [system test response](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/_dev/deploy/docker/files/intelligence-api-config.yml#L69C452-L69C466)), and the CEL also [reserializes](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/data_stream/intelligence/agent/stream/cel.yml.hbs#L59) into JSON [as an integer](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/data_stream/intelligence/sample_event.json#L58).

This raises some questions:
* What JSON is being sent to Elasticsearch in such cases?
* Why is this happening for relatively small integers?
* Is there an issue in cel-go or the CEL input or Beats that could be fixed to improve this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.