apache / apache/datafusion

Provide a way to enable source level statistics for tables registered in the CLI

Open
#3,774 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
#1347 enabled collection of statistics by default on the `ListingOptions` constructor, though the tables created with `CREATE EXTERNAL TABLE` can't still use this feature since they are created manually.
https://github.com/apache/arrow-datafusion/blob/e54110fb592e03704da5f6ebd832b8fe1c51123b/datafusion/core/src/execution/context.rs#L486-L488

**Describe the solution you'd like**
We already have a per file extension listing option implementation for the `read_` dataframe APIs (e.g. `CsvReadOptions`, `ParquetReadOptions`) and they have sane defaults (like `collect_stats` is `false` for CSV and `true` for Parquet). I wonder whether we can just use them here and obtain the `ListingOptions` directly from them.

**Describe alternatives you've considered**
Leaving as is, or enabling them globally (instead of refactoring that part to use `ReadOptions`) by just setting the flag to true.

Contributor guide

Open the contributing guide

Research direction

Start at the ListingOptions construction in datafusion/core/src/execution/context.rs around the linked lines, then compare it with the existing CsvReadOptions and ParquetReadOptions implementations. Determine how CLI-created external tables can obtain per-format listing options, with completion marked by source-level statistics following the formats' established collect_stats defaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
cli, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.