Provide a way to enable source level statistics for tables registered in the CLI
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
#1347 enabled collection of statistics by default on the `ListingOptions` constructor, though the tables created with `CREATE EXTERNAL TABLE` can't still use this feature since they are created manually.
https://github.com/apache/arrow-datafusion/blob/e54110fb592e03704da5f6ebd832b8fe1c51123b/datafusion/core/src/execution/context.rs#L486-L488
**Describe the solution you'd like**
We already have a per file extension listing option implementation for the `read_` dataframe APIs (e.g. `CsvReadOptions`, `ParquetReadOptions`) and they have sane defaults (like `collect_stats` is `false` for CSV and `true` for Parquet). I wonder whether we can just use them here and obtain the `ListingOptions` directly from them.
**Describe alternatives you've considered**
Leaving as is, or enabling them globally (instead of refactoring that part to use `ReadOptions`) by just setting the flag to true.
Contributor guide
Research direction
Start at the ListingOptions construction in datafusion/core/src/execution/context.rs around the linked lines, then compare it with the existing CsvReadOptions and ParquetReadOptions implementations. Determine how CLI-created external tables can obtain per-format listing options, with completion marked by source-level statistics following the formats' established collect_stats defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- cli, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100