Statistic: data_size should be in ColumnStatistics.
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
Current, we use `total_byte_size` to store whole byte_size.
But it isn't good enough, a better way is to put `avg_data_size/total_date_size` into `ColumnStatistics`.
a `total_byte_size` of statistic is useless, because we hard to propagate it in `Statistic derive`.
But if we use `Column avg_data_size` we can use it to propagate it into other Plan.
Spark:
```scala
case class ColumnStat(
....
avgLen: Option[Long] = None,
maxLen: Option[Long] = None,
...
```
Presto
```java
public final class ColumnStatistics
{
private final Estimate nullsFraction;
private final Estimate distinctValuesCount;
private final Estimate dataSize;
private final Optional range;
}
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by locating ColumnStatistics and the Statistic derive logic, then trace how total_byte_size is currently used when statistics are propagated into other plans. The issue is done when column-level average or total data size is represented in ColumnStatistics and can be propagated through the relevant planning statistics flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100