elastic / elastic/integrations
Add wired streams (logs.ecs) support to all eligible input-type packages
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Summary
Add `use_logs_stream` support to all eligible input-type packages, allowing users to write directly to the `logs.ecs` wired streams endpoint.
### Motivation
[Wired streams](https://www.elastic.co/docs/solutions/observability/streams/wired-streams) are a centralized way to manage log data in Elastic. When input packages write to the `logs.ecs` endpoint, users gain access to several powerful capabilities without making any changes to their agent configuration:
- **Partitioning multiplexed data into child streams.** Input packages like `cel`, `gcp_pubsub`, `http_endpoint`, and `kafka_log` often carry multiplexed data — a single input source containing many different log types. With wired streams, users can [partition](https://www.elastic.co/docs/solutions/observability/streams/management/partitioning) this data into meaningful child streams (e.g. `logs.ecs.firewall`, `logs.ecs.auth`) based on field values, with different retention policies per child stream. This routing is configured entirely in Kibana — no agent-side changes required.
- **AI-assisted partitioning.** Streams can [analyze incoming data and suggest partitions](https://www.elastic.co/docs/solutions/observability/streams/management/partitioning#streams-AI-partitioning) automatically using a Generative AI connector, making it easy to organize data without manual field inspection.
- **Stream processing.** Each stream supports [field extraction](https://www.elastic.co/docs/solutions/observability/streams/management/extract) and [schema mapping](https://www.elastic.co/docs/solutions/observability/streams/management/schema), giving users per-stream control over how data is parsed and indexed.
- **Hierarchical inheritance.** Wired streams support a parent-child hierarchy where child streams automatically inherit mappings, lifecycle settings, and processors from their parent, keeping configuration consistent.
Input-type packages are the natural fit for wired streams because they pass through raw data without transformation — the processing happens in Streams rather than in an ingest pipeline.
### Reference Implementation
The `filestream` package (v2.4.0, PR #17909) is the reference. The pattern is:
**Agent template:**
```handlebars
{{#if use_logs_stream}}
index: logs.ecs
{{else}}
data_stream:
dataset: {{data_stream.dataset}}
{{#if pipeline}}
pipeline: {{pipeline}}
{{/if}}
{{/if}}
```
**Manifest variable:**
```yaml
- name: use_logs_stream
type: bool
title: Use the "logs.ecs" data stream
description: |
When enabled, data ingested by this integration is written to the "logs.ecs" data stream.
**The 'Ingest Pipeline' and the configured 'Dataset name' are ignored**. You also need to
Turn on wired streams in Streams Settings and to enable Allow agents to write to Streams
for the output policy in Fleet Settings.
required: false
show_user: true
default: false
```
### Eligible Packages (11)
Each package needs three changes: agent template conditional, manifest variable + version bump, and changelog entry.
| Package | Current Kibana Constraint | Notes |
|---------|--------------------------|-------|
| `cel` | `^8.19.0 \|\| ^9.1.0` | `pipeline:` is far from `data_stream:` — both need wrapping |
| `http_endpoint` | `^8.15.0 \|\| ^9.0.0` | Straightforward |
| `journald` | `^8.8.0 \|\| ^9.0.0` | No `pipeline` variable |
| `gcp_pubsub` | `^8.13.0 \|\| ^9.0.0` | Straightforward |
| `azure_blob_storage` | `^8.16.0 \|\| ^9.0.0` | Straightforward |
| `google_cloud_storage` | `^8.13.0 \|\| ^9.0.0` | Straightforward |
| `websocket` | `^8.16.3 \|\| ^8.17.1 \|\| ^9.0.0` | `pipeline:` is far from `data_stream:` |
| `windows_etw` | `^8.17.0 \|\| ^9.0.0` | `condition:` line must remain outside the conditional |
| `winlog` | `^8.10.1 \|\| ^9.0.0` | `condition:` line must remain outside the conditional |
| `azure_logs` | `^8.19.10 \|\| ~9.1.10 \|\| ~9.2.4 \|\| ^9.3.0` | `pipeline:` appears before `data_stream:` |
| `kafka_log` | `^8.13.0 \|\| ^9.0.0` | — |
### In Progress (2)
`tcp` (PR #18018) and `udp` (PR #18017) already have `use_logs_stream` but write to the deprecated `logs` index instead of `logs.ecs`.
### Ineligible Packages
| Package | Reason |
|---------|--------|
| `log` | Deprecated. Users should migrate to `filestream`. |
| `unifiedlogs` | Ships a built-in ingest pipeline that parses raw NDJSON from the macOS `log` CLI into ECS and `unified_log.*` fields. Without this pipeline the data is not usable. This makes it a processing integration rather than a pass-through input, and incompatible with wired streams which bypass ingest pipelines. |
### 8.x Compatibility Concern
The `filestream` package set its Kibana constraint to `^9.4.0` when it added this feature, but `filestream` had already dropped 8.x support. All 11 eligible packages listed above still support 8.x stacks. Elasticsearch 8.x is a supported platform and we cannot drop support for it.
Bumping the Kibana constraint to `^9.4.0` would cut off 8.x users from receiving **any** future updates to these packages — not just wired streams, but any bugfix or enhancement.
The `use_logs_stream` variable defaults to `false`, so the template change is harmless on 8.x — the standard `data_stream:` block renders as before. The risk is that a user on 8.x could see the toggle, enable it, and get an error because the `index: logs.ecs` directive is not supported by 8.x agents.
**Options to explore:**
1. **Keep the existing 8.x Kibana constraint and document the requirement.** Add a note to the `use_logs_stream` variable description stating it requires Kibana 9.4+. Users on 8.x would see the toggle but enabling it would fail. This is the simplest approach but has poor UX for 8.x users.
2. **Investigate conditional variable visibility based on stack version.** If Fleet/Kibana could hide variables based on the running stack version, 8.x users would never see the toggle. This would be the ideal solution but is out of scope for this issue and may require Fleet changes.
3. **Maintain a parallel branch for 8.x releases.** Keep a long-lived `8.x` branch and backport bugfixes from `main` into it. This preserves the ability to ship 8.x updates while allowing `main` to move to `^9.4.0`. This is operationally painful — we do not maintain version branches today — and adds ongoing maintenance burden for every future change.
~A solution must be identified before implementation begins.~ I talked with a few folks about this, and option 1 should be fine for this. The UDP/TCP implementations already mention requiring 9.4 in their variable descriptions.
### Related
- Wired streams docs: https://www.elastic.co/docs/solutions/observability/streams/wired-streams
- Partitioning docs: https://www.elastic.co/docs/solutions/observability/streams/management/partitioning
- `filestream` implementation: #17909
- `tcp` fix: #18018
- `udp` fix: #18017
Contributor guide
Assessment
This issue has not been assessed yet.