elastic / elastic/roadmap

ES|QL Data Federation: Query files in Amazon S3

Open
#307 1 comment 0 reactions 1 assignee Claimed by @tylerperk View on GitHub
Component: Elasticsearch product-area:observability product-area:platform product-area:search product-area:security v9.5.0
Dominant language
No language data
Stars
6
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Value Proposition

Not all enterprise data lives in Elasticsearch. Historical logs, compliance archives, security reference data, and partner feeds accumulate in object storage because moving them is expensive or impractical. Without the ability to query this data from within Elasticsearch, every query works with an incomplete picture — requiring a second query engine, a separate interface, and manual reconciliation of results.

ES|QL Data Federation extends Elasticsearch to query data in external sources directly, with no ingestion and no second query engine. Register a data source, name a dataset, and query it with the same `FROM` clause and the same ES|QL pipeline you use for indexed data. A dataset behaves like a virtual index — same syntax, same Kibana interface, nothing new to learn.

## Expected Outcome

Ships as Experimental with Amazon S3 as the first supported object store, covering the dominant share of enterprise data in cloud storage.

- Datasets as virtual indices — queried via `FROM dataset_name` with the full ES|QL processing pipeline (WHERE, STATS, EVAL, SORT, LIMIT, LOOKUP JOIN, etc.)
- Parquet, NDJSON, and CSV/TSV formats with automatic schema inference
- gzip and zstd compression support for text formats
- Filter pushdown, column projection, and LIMIT pushdown for query performance
- A single query can combine Elasticsearch indices and external datasets in one `FROM` clause
- Kibana CRUD UI for managing data sources and datasets
- RBAC via standard Elasticsearch index privileges on dataset names
- Encrypted credential management (stored at rest in cluster state, decrypted only at query time)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.