Dealing with huge datasets, using the Elasticsearch Sample and Diversified Sampler Aggregation as a prefilter
- Dominant language
- Python
- Stars
- 693
- Forks
- 112
- PR merge metrics
- No merged PRs in 30d
Description
Hi!
it would be cool to be able to use the `sampler` aggregation like this:
```python
df['agent.name'].es_sampler(1000).value_counts()
```
In environments, e.g. the one I am currently working on, we receive up to 2.5 TB of data every 30 minutes. So querying the entire documents for 100% accuracy is not worth the cpu%. The sampler would allow us to look at a subset of that adequately and comprehensively.
Contributor guide
Research direction
Start by tracing the DataFrame expression `df['agent.name']` and its `value_counts()` path, then inspect how Elasticsearch aggregations are represented. Compare that path with Elasticsearch's Sample and Diversified Sampler Aggregations; done means a sampler-based prefilter can be requested for this workflow with the proposed `es_sampler(1000)` usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100