apache / apache/datafusion

Add reservoir sampling

Open
#11,554 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We have a large sample of statistical data. All we need is a subset of the data that maintains statistical significance while being able to return a much smaller result to users since insignificantly small values aren't contained resulting in much lower latency.

### Describe the solution you'd like

Add the ability to (statistically) sample rows. We've done this using reservoir sampling before. I imagine statistical sampling is a widely enough used function that it should be supported first-class.

### Describe alternatives you've considered

I don't know enough about DataFusion to know whether this is possible via a UDF. In the past, we've had issues where records pushed into the query layer are sampled. However, the underlying record is still held onto as immediately materializing it would result in tiny and inefficient 1-row records, but eventually, they need to be materialized as otherwise memory explodes.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing DataFusion's query execution and UDF capabilities, then clarify the sampling semantics, materialization behavior, and supported query interface before defining the implementation scope and acceptance tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.