elastic / elastic/roadmap

Query Streams and Draft Mode

Open
#149 0 comments 0 reactions 1 assignee Claimed by @LucaWintergerst View on GitHub
Component: Elasticsearch Component: Kibana product-area:observability
Dominant language
No language data
Stars
6
Forks
1
PR merge metrics
No merged PRs in 30d

Description

### Value proposition:

Users need more flexibility in how they organize, view, and process their data. Sometimes they need to create custom, reusable views that span multiple data sources (e.g., "all production errors"), or filter a single source.

Other times, they need to safely test new ingest-time processing and partitioning rules without risking production data. This feature introduces two powerful capabilities:
- Query Streams let users define new virtual, read-only streams using ES|QL
- Draft Mode provides a way to test and preview new ingest logic at query-time before making it permanent.

### Expected behaviors:

This feature is composed of two related functionalities:

**1. Query Streams (Virtual Data Views)**
* Define and create a new, queryable stream using a free-form **ES|QL query**.
* Create "standalone" streams to unify data from multiple sources (e.g., `FROM logs.nginx, logs.k8s.nginx*`).
* Create "sub-streams" to create a filtered, read-only view of an existing stream (e.g., `FROM my_stream WHERE status_code > 400`).
* These streams are **virtual and read-only**; they do not store data and resolve the query at read time.
* Attach assets like dashboards and alerts directly to these virtual streams once available in Streams (outside of this initial feature).

**2. Draft Mode (Safe Processing Testing)**
* Create new Log Sub Streams in a new **"Draft Mode."**
* Define partitioning and processing rules in this safe "sandbox" environment.
* This logic is **applied at query time**, not at ingest time, allowing you to preview the results on live data without any permanent impact.
* Validate your new processing logic (e.g., in dashboards) to ensure it's correct.
* Once confident, **"Promote"** the draft rules to make them permanent, which applies them to the live ingest pipeline.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.