elastic / elastic/elastic-serverless-forwarder
Document options for enriching events when using ESF
- Dominant language
- Python
- Stars
- 38
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the enhancement:**
We need to add some info to the ESF docs to help customers who want to use Beats-style processors, but cannot because ESF does not support them. We should explain that they can use a custom ingest pipeline instead and provide examples.
**Describe a specific use case for the enhancement or feature:**
Customers who are migrating away from functionbeat to ESF might have a use case that requires processors. For example, they might be using processors like `decode_json_fields`, `add_fields`, and `add_cloud_metadata` to enrich their data. However ESF does not support Beats-style processors and may not have access to all the fields needed to enrich events with cloud metadata.
Specifically, we need to:
* Add some clarification to the docs about processor operations that are supported by ingest pipelines.
For functionalities like adding/removing/renaming fields, JSON, and more, ESF leverages the processors available in the ingest pipeline on Elasticsearch. The integrations allow users to customize the default pipelines by using a custom pipeline. We need to explain this and point to the documentation that describes how to use custom pipelines. (There's a tutorial [here](https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html), but there's probably something even better in the Elasticsearch docs.)
* Note that ESF is an AWS Lambda function, so it only supports the metadata applicable in this execution environment. Given the typical content of a cloud field:
```
{
"cloud": {
"account.id": "123456789012",
"availability_zone": "us-east-1c",
"instance.id": "i-4e123456",
"machine.type": "t2.medium",
"image.id": "ami-abcd1234",
"provider": "aws",
"region": "us-east-1"
}
}
```
Here is the same field from ESF:

All the other cloud fields (`image.id`, `availability_zone`, `machine.type`, and `instance.id`) are specific to EC2 and not applicable to a Lambda runtime. Even if AWS probably runs these functions on some EC2-based compute unit, it is an implementation detail they don’t make available to customers.
**Contacts**
@zmoog
Contributor guide
Assessment
This issue has not been assessed yet.