ES|QL Data Federation: Query files in Amazon S3
- Dominant language
- No language data
- Stars
- 6
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Value Proposition
Not all enterprise data lives in Elasticsearch. Historical logs, compliance archives, security reference data, and partner feeds accumulate in object storage because moving them is expensive or impractical. Without the ability to query this data from within Elasticsearch, every query works with an incomplete picture — requiring a second query engine, a separate interface, and manual reconciliation of results.
ES|QL Data Federation extends Elasticsearch to query data in external sources directly, with no ingestion and no second query engine. Register a data source, name a dataset, and query it with the same `FROM` clause and the same ES|QL pipeline you use for indexed data. A dataset behaves like a virtual index — same syntax, same Kibana interface, nothing new to learn.
## Expected Outcome
Ships as Experimental with Amazon S3 as the first supported object store, covering the dominant share of enterprise data in cloud storage.
- Datasets as virtual indices — queried via `FROM dataset_name` with the full ES|QL processing pipeline (WHERE, STATS, EVAL, SORT, LIMIT, LOOKUP JOIN, etc.)
- Parquet, NDJSON, and CSV/TSV formats with automatic schema inference
- gzip and zstd compression support for text formats
- Filter pushdown, column projection, and LIMIT pushdown for query performance
- A single query can combine Elasticsearch indices and external datasets in one `FROM` clause
- Kibana CRUD UI for managing data sources and datasets
- RBAC via standard Elasticsearch index privileges on dataset names
- Encrypted credential management (stored at rest in cluster state, decrypted only at query time)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.