Large integers from CEL arrive in ES as Doubles
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
Elasticsearch is okay with integers like `235549249` and larger, and can convert them to strings.
Expand for an Elasticsearch example
```
PUT _ingest/pipeline/test_convert
{
"description": "Pipeline to test conversion of large integers",
"processors": [
{
"convert": {
"field": "small",
"target_field": "small_string",
"type": "string"
}
},
{
"convert": {
"field": "large",
"target_field": "large_string",
"type": "string"
}
}
]
}
POST _ingest/pipeline/test_convert/_simulate
{
"docs": [
{
"_source": {
"small": 12345,
"large": 2355492490000000000
}
}
]
}
```
```
{
"docs": [
{
"doc": {
"_index": "_index",
"_version": "-3",
"_id": "_id",
"_source": {
"small": 12345,
"large": 2355492490000000000,
"small_string": "12345",
"large_string": "2355492490000000000"
},
"_ingest": {
"timestamp": "2025-04-03T08:58:03.968461461Z"
}
}
}
]
}
```
However, when receiving such a number from Beats (at least from the CEL input), it is of type `Double`.
The [convert processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/convert-processor.html) will produce the string `"2.35549249E8"`. Trying to convert to an integer before converting to a string fails with an error message saying it can't convert the string "2.35549249E8" (so, the convert processor doesn't natively convert from Double to integer).
Expand for a script processor that does a type check and conversion in Painless
```
- script:
description: Convert large integer to a string without an exponent
lang: painless
source: |
def value = (Object) ctx.json.id;
if (value instanceof Integer) {
ctx.json["id_type_name"] = "Integer";
} else if (value instanceof Double) {
ctx.json["id_type_name"] = "Double";
} else if (value instanceof String) {
ctx.json["id_type_name"] = "String";
} else {
ctx.json["id_type_name"] = "Unknown";
}
ctx.json.id = Long.toString((long) ctx.json.id);
```
Result:
```
json.id_type_name = "Double"
json.id = 235549249
```
This requires special handling, such as [in the `ti_anomali` integration](https://github.com/elastic/integrations/blob/703b76e5b9004bc2c9c06e5844e6a56465b0cb4c/packages/ti_anomali/data_stream/intelligence/elasticsearch/ingest_pipeline/default.yml#L524-L529). In that case, notice that the original server response is an integer in JSON (e.g. `"id":235548914` in the [system test response](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/_dev/deploy/docker/files/intelligence-api-config.yml#L69C452-L69C466)), and the CEL also [reserializes](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/data_stream/intelligence/agent/stream/cel.yml.hbs#L59) into JSON [as an integer](https://github.com/elastic/integrations/blob/b8605a5b619afea5a3b10e4eb32ff077ce96fafd/packages/ti_anomali/data_stream/intelligence/sample_event.json#L58).
This raises some questions:
* What JSON is being sent to Elasticsearch in such cases?
* Why is this happening for relatively small integers?
* Is there an issue in cel-go or the CEL input or Beats that could be fixed to improve this?
Contributor guide
Assessment
This issue has not been assessed yet.