elastic / elastic/elastic-agent
Convert input streams into individual inputs at runtime and in health reporting
- Dominant language
- Go
- Stars
- 275
- Forks
- 264
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 298
Description
Elastic Agent supports inputs with multiple streams. In all cases the data shipped is exactly the same if multiple copies of the same input were created instead. The `streams` list is a short cut to avoid repetition in the configuration.
For example, the configuration below uses `streams` to define a single `system/metrics` input with multiple metricsets and a single `log` input with multiple streams. These configurations are passed directly to the underlying Beat where in the case of Filebeat the [streams are expanded](https://github.com/elastic/beats/blob/a73a1f580b55e3c6fc2d491cd27533634ec5cd2e/x-pack/libbeat/management/generate.go#L85-L88) into individual Beat inputs in the Beat. Metricbeat natively supports the concept of a module with multiple metricsets.
```yaml
inputs:
- id: unique-system-metrics-input
type: system/metrics
use_output: default
data_stream:
namespace: default
streams:
- id: unique-system-metrics-input-cpu
data_stream:
dataset: system.cpu
metricsets:
- cpu
- id: unique-system-metrics-input-mem
data_stream:
dataset: system.memory
metricsets:
- memory
- id: logfile-apache-347de5f6-bfc5-4428-9969-b809582bef75
type: logfile
use_output: default
data_stream:
namespace: default
streams:
- id: logfile-apache.access-347de5f6-bfc5-4428-9969-b809582bef75
data_stream:
dataset: apache.access
paths:
- /var/log/apache2/access.log*
- id: logfile-apache.error-347de5f6-bfc5-4428-9969-b809582bef75
data_stream:
dataset: apache.error
paths:
- /var/log/apache2/error.log*
```
This works well as a convenience for writing configuration, but the `streams` concept was never modelled in the agent health and [component model](https://github.com/elastic/elastic-agent/blob/main/docs/architecture.md). As a work around, https://github.com/elastic/beats/pull/39209 began reporting the individual stream state in the free form `payload` JSON object in the control protocol and aggregating the stream state into the overall input state. This correctly shows which integration is unhealthy in the Fleet UI but does not correctly show the affected stream as the Fleet UI also does not model the streams concept in the input health reporting UI.
For example, the `system/metrics` input above is reported as follows when Elastic Agent checkins in with Fleet:
```json
"components": [
{
"id": "system/metrics-default",
"name": "system/metrics",
"state": 2,
"message": "Healthy: communicating with pid '88252'",
"units": [
{
"unit_id": "system/metrics-default",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "system/metrics-default-unique-system-metrics-input",
"unit_type": 0,
"state": 2,
"message": "Healthy",
"payload": {
"streams": {
"unique-system-metrics-input-cpu": {
"error": "",
"status": "HEALTHY"
},
"unique-system-metrics-input-mem": {
"error": "",
"status": "HEALTHY"
}
}
}
}
]
}
]
```
Rather than change the control protocol and checkin API to understand streams, it is simpler to automatically convert each stream into an instance of the input it is associated with. This maintains the configuration syntax, fixes the health reporting issues, and likely requires no change to any system component except the Elastic Agent configuration translation. This would introduce a small breaking change for standalone agent configurations that use streams without must now specify those streams with unique IDs (Fleet already provides a unique ID per stream).
The health of the `system/metrics` example would instead be reported as something like:
```json
"components": [
{
"id": "system/metrics-default",
"name": "system/metrics",
"state": 2,
"message": "Healthy: communicating with pid '88252'",
"units": [
{
"unit_id": "system/metrics-default",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "system/metrics-default-unique-system-metrics-input-cpu",
"unit_type": 0,
"state": 2,
"message": "Healthy",
}
{
"unit_id": "system/metrics-default-unique-system-metrics-input-mem",
"unit_type": 0,
"state": 2,
"message": "Healthy",
}
]
}
]
```
This change could be done for as part of the beat receivers work as Elastic Agent has already assumed the job of translating streams to beat inputs when generating the collector pipeline for beat receivers:
https://github.com/elastic/elastic-agent/blob/a8bbcf66d38f2941587611f4ba6db3b2494be16d/internal/pkg/otel/translate/otelconfig.go#L505-L518
### Acceptance Criteria
- [ ] Confirm with a quick prototype or PoC that the Fleet input health UI continues to work without change if streams are reported as inputs. If a change is required, determine which changes and their cost.
- [ ] Document the user facing breaking change for standalone agent configurations that all streams will require unique IDs. The changes to the output of `elastic-agent status` are documented.
- [ ] New tests are added proving the streams to input translation works without issue.
- [ ] Existing tests based on health status continue to pass, particularly those for existing integrations.
Contributor guide
Assessment
This issue has not been assessed yet.