elastic / elastic/package-spec

[Change Proposal] Add explicit mechanism to share OTel collector extensions

Open
#1,187 0 comments 1 reaction 0 assignees View on GitHub
discuss
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

Configuration templates for `otelcol` inputs can contain [extensions](https://opentelemetry.io/docs/collector/components/extension/). They are namespaced as any other component, so they can only be used as part of a single package policy.

Extensions are usually expected to be shared and referenced from different resources, so they are defined once, and used in many places. With the current implementation they can only be shared in the same policy template where they are defined.

Define an explicit mechanism to be able to define extensions and use them in different package policies in the same agent policy.

This will probably imply the following changes to be defined:
* A mechanism in input packages to announce that they define OTel extensions intended to be shared, and of what kind.
* A new type of variable that can be used in OTel input and integration packages to reference extensions available in the agent policy.

Follow up of https://github.com/elastic/kibana/pull/273374#issuecomment-4724051996.

---

A possible design could be the following:

An input package could contain policy templates with a new type called `extension`, like this:
```yaml
policy_templates:
- name: awscredentialsprovider
type: extension
title: AWS Credentials Provider OTel Extension
description: Configures an OTel extension with AWS credentials.
input: otelcol
vars:
...
```
Its config template would contain only the extension:
```yaml
extensions:
awscredentialsprovider:
credentials:
access_key_id: {{access_key_id}}
secret_access_key: {{secret_access_key}}
assume_role:
arn: {{role_arn}}
external_id: {{external_id}}
```

Then an input or integration package can reuse it by defining a new type of variable.
```yaml
vars:
- name: auth_extension
type: otel_extension
title: Authentication extension
required: true
```
And use it in the template:
```yaml
receivers:
awscloudwatch:
region: us-east-1
credentials_provider: {{auth_extension}}
```

With this design Fleet would need to do the following:
* When a policy with the `extension` type is used in a policy, it needs to assign a unique component id, the same extension could be defined several times in the same agent policy.
* When a variable of type `otel_extension` is used, it must contain as value a reference to the policy with the extension. API could use the package policy id, UI could offer a selector with the list of package policies with shared extensions in the agent policy, and translate to the policy id under the hood.
* When Fleet renders the configuration template using the `otel_extension` variable, it is populated with the unique component id of the extension in the extension policy.
* The resulting configuration with the samples above would be something like the following:
```yaml
extensions:
awscredentialsprovider/per_policy_unique_id:
credentials:
access_key_id: "some_access_key"
secret_access_key: {{secret_access_key}}
assume_role:
arn: "arn::some_arn"
external_id: "some_external_id"
receivers:
awscloudwatch/some-uuid:
region: us-east-1
credentials_provider: "awscredentialsprovider/per_policy_unique_id"
...
awscloudwatch/other-uuid:
region: eu-east-1
credentials_provider: "awscredentialsprovider/per_policy_unique_id"
...
```
Notes:
* This model doesn't validate that the extension used is of the expected kind, this can only be ensured on runtime. We could allow to define types of extensions, but it can be difficult and restrictive to maintain a catalog of them.
* A policy template of the type `extension` must only produce the configuration of an extension, so there is no ambiguity on what extension is shared with other policies.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing how package policy templates and OTel input configurations are represented and rendered, then review the linked Kibana pull request discussion for context. Done means the package specification defines shared extension policies and references, including unique component IDs across policies.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, yaml
Domain
observability-sre, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.