elastic / elastic/elastic-serverless-forwarder

Document options for enriching events when using ESF

Open
#324 5 comments 1 reaction 1 assignee Claimed by @zmoog View on GitHub
documentation Team:Docs
Dominant language
Python
Stars
38
Forks
50
PR merge metrics
No merged PRs in 30d

Description

**Describe the enhancement:**

We need to add some info to the ESF docs to help customers who want to use Beats-style processors, but cannot because ESF does not support them. We should explain that they can use a custom ingest pipeline instead and provide examples.

**Describe a specific use case for the enhancement or feature:**

Customers who are migrating away from functionbeat to ESF might have a use case that requires processors. For example, they might be using processors like `decode_json_fields`, `add_fields`, and `add_cloud_metadata` to enrich their data. However ESF does not support Beats-style processors and may not have access to all the fields needed to enrich events with cloud metadata.

Specifically, we need to:

* Add some clarification to the docs about processor operations that are supported by ingest pipelines.

For functionalities like adding/removing/renaming fields, JSON, and more, ESF leverages the processors available in the ingest pipeline on Elasticsearch. The integrations allow users to customize the default pipelines by using a custom pipeline. We need to explain this and point to the documentation that describes how to use custom pipelines. (There's a tutorial [here](https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html), but there's probably something even better in the Elasticsearch docs.)

* Note that ESF is an AWS Lambda function, so it only supports the metadata applicable in this execution environment. Given the typical content of a cloud field:

```
{
"cloud": {
"account.id": "123456789012",
"availability_zone": "us-east-1c",
"instance.id": "i-4e123456",
"machine.type": "t2.medium",
"image.id": "ami-abcd1234",
"provider": "aws",
"region": "us-east-1"
}
}

```

Here is the same field from ESF:

![image](https://user-images.githubusercontent.com/14206422/230444472-ae311080-7cc7-406f-9db9-957d63ec9543.png)

All the other cloud fields (`image.id`, `availability_zone`, `machine.type`, and `instance.id`) are specific to EC2 and not applicable to a Lambda runtime. Even if AWS probably runs these functions on some EC2-based compute unit, it is an implementation detail they don’t make available to customers.

**Contacts**
@zmoog

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.