apache / apache/datafusion

Feature: Support Sample

Open
#16,533 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The TABLESAMPLE statement is used to sample the table.

Different DBs have different sample implementations.

Spark:

https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-sampling.html

example with replacement, poisson sample.
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.sample.html

it implements
1. Sample the table down to the given number of rows.
2. Sample the table down to the given percentage.
a. poisson sample, (only in dataframe api)
b. bernoulli sample.

Spark introduced a Sample logical plan, and many other dataframe apis are also based on this logical plan. e.g.
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.randomSplit.html

Hive:
https://cwiki.apache.org/confluence/display/hive/languagemanual+sampling

1. Sample the table down to the given number of rows.
2. Sample the table down to the given percentage. (bernoulli)
3. Sample on column, it's useful for clustered table.

Clickhouse:

https://clickhouse.com/docs/sql-reference/statements/select/sample

1. Sample the table down to the given number of rows.
2. Sample the table down to the given percentage.
3. Sample with offset.

Postgres:

https://wiki.postgresql.org/wiki/TABLESAMPLE_Implementation
1. BERNOULLI sample.
2. SYSTEM

### Describe the solution you'd like

Add a Sample logical plan.

### Describe alternatives you've considered

I have considered resusing current logical plan, e.g. Filter. But it seems that it's hard to implement poisson sample with current logical plan.
in spark, real_seed = input_seed + partition_id. then different partitions have different sample results. it makes sense to me. it's also hard to implement with current logical plan.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by comparing the linked Spark, Hive, ClickHouse, and PostgreSQL sampling semantics with DataFusion's existing logical-plan design. Define the Sample logical plan and determine how row-count, percentage, replacement, offset, and seed behavior should be represented; done means the supported sampling forms are implemented and covered by appropriate tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.