elastic / elastic/eland

Dealing with huge datasets, using the Elasticsearch Sample and Diversified Sampler Aggregation as a prefilter

Open
#494 0 comments 0 reactions 0 assignees View on GitHub
enhancement topic:dataframe
Dominant language
Python
Stars
693
Forks
112
PR merge metrics
No merged PRs in 30d

Description

Hi!

it would be cool to be able to use the `sampler` aggregation like this:

```python
df['agent.name'].es_sampler(1000).value_counts()
```

In environments, e.g. the one I am currently working on, we receive up to 2.5 TB of data every 30 minutes. So querying the entire documents for 100% accuracy is not worth the cpu%. The sampler would allow us to look at a subset of that adequately and comprehensively.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the DataFrame expression `df['agent.name']` and its `value_counts()` path, then inspect how Elasticsearch aggregations are represented. Compare that path with Elasticsearch's Sample and Diversified Sampler Aggregations; done means a sampler-based prefilter can be requested for this workflow with the proposed `es_sampler(1000)` usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.