elastic / elastic/package-spec

[Change Proposal] Attribute-based auto-installation of integrations

Open
#1,202 10 comments 1 reaction 0 assignees View on GitHub
discuss Team:Fleet
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

## Context

The auto-installation of integrations is based on the specification of a dataset. If the dataset exists and data is present, the matching integration will be installed. Integrations specify this in the `discovery` section of the `manifest.yml`file. To provide an example, the [OpenTelemetry Collector Internal Telemetry](https://github.com/elastic/integrations/blob/1a24f6a3e408b0f8096c06e2995177c36ed89272/packages/otel_collector_internal_telemetry/manifest.yml#L18-L20) integration defines:

```
discovery:
datasets:
- name: collectortelemetry.otel
```

OpenTelemetry data ingested by the SDKs, by default, is routed to data streams
- `logs-generic.otel-default`
- `metrics-generic.otel-default`
- `traces-generic.otel-default`

and the data streams to the `generic.otel` dataset.

New OpenTelemetry content-only integrations (also referred to as content packs) are often language and technology specific. The following scenarios are examples to make it more tangible.

### Scenario 1

OpenTelemetry data from a Java application is ingested through the managed OTLP endpoint. The `generic.otel` dataset is available and data is present. Matching based on the dataset `generic.otel` installs the `Android OpenTelemetry Assets` and the `RUM OpenTelemetry Assets` automatically. The user is never ingesting OpenTelemetry Android or RUM data, the automatically installed dashboards have no data to show.

### Scenario 2

Prometheus data from a Node.js application is ingested through the [Prometheus remote write endpoint](https://www.elastic.co/docs/manage-data/data-store/data-streams/tsds-ingest-prometheus-remote-write#send-data-to-different-data-streams). The `generic.prometheus` dataset is available and data is present. A technology-specific Java SpringBoot integration is automatically installed because the data stream is available and data is present. The dashboard will not find any data.

## Change

It should be possible to define attributes and attribute values to match in order to express the auto-installation condition in more detail. Something like:

```
discovery:
datasets:
- name: generic.otel
attributes:
resource.attributes.telemetry.sdk.language: java
resource.attributes.telemetry.sdk.name: android
scope.name: io.opentelemetry.okhttp-3.0
```

@juliaElastic mentioned this change impacts the package spec and Kibana.

## Open

Data can be routed to custom datasets ([PRW](https://www.elastic.co/docs/manage-data/data-store/data-streams/tsds-ingest-prometheus-remote-write#send-data-to-different-data-streams), [OTel](https://www.elastic.co/docs/reference/opentelemetry/data-streams#managed-otlp-endpoint)). IIUC the auto-installation mechanism would not install an integration, because of the custom name of the dataset. To mitigate this issue, should it be possible to omit the dataset definition and only provide attributes and attribute values to match? An integration could be installed if the attributes and values match independent of the dataset.

Contributor guide

Open the contributing guide

Research direction

Start with the discovery section in the example manifest.yml and review the package specification and Kibana impact mentioned in the issue. Clarify how attribute matching should interact with dataset names, including custom datasets and attribute-only rules. Done means the matching semantics and required package-spec and Kibana changes are agreed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.