Process very large data and result sets with ES|QL
- Dominant language
- No language data
- Stars
- 6
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
**Value Proposition**
Users with large datasets sometimes need to return more rows from a query than ES|QL allows. We're currently constrained to the amount of available Java heap memory as we lack features like Spill to Disk. Without pagination, increasing the LIMIT just creates a larger response object putting more stress on the network and the client, such as Kibana. We need to be able to return results in pages (say, 10 at a time for a Search results web page) without forcing a rerun of the same query.
Also, as much as we love Kibana, not all analysis happens there. Sometimes you need to share a large number of rows from a query with an external party (auditor, federal agency) that doesn't have access to your cluster. Users have asked to be able to process and export hundreds of thousands, even millions, of rows from a query.
**Expected Outcomes**
Features like pagination and spill to disk would enable ES|QL to handle and return much larger working sets, enabling higher LIMITs and larger exports to external systems. Pagination facilitates efficiently browsing through large result sets in Kibana Discover and in search use cases.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.