Column prune related improvements
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Enhancement
Column prune will improve performance in several aspects:
- Reduce memory usage, since fewer column data may be loaded.
- Reduce network overhead, because we have dependant strorage systems: tikv, tiflash, fewer columns, fewer data will be transfered
- Reduce CPU usage, some operators' workload are related with the size of their input columns. For example, PhysicalExchange node will partition all its input columns, therefore, fewer columns, fewer workload.
However, there are still several column prune related issues in tidb now:
1. Useless columns in union operator may not be pruned
Currently, when a LogicalUnionAll operator's parent operator doesn't use the LogicalUnionAll operator's output columns(e.g. select count(*) from (xx union xx)), these useless columns won't be pruned:
https://github.com/pingcap/tidb/blob/b96f08134d66196ca760a6caf4d0e9cd044fff97/pkg/planner/core/rule_column_pruning.go#L303-L309
2. Useless columns used in filter expressions won't be pruned
When column prune optimization is executed, filter is still included in DataSource operator. Currently, the DataSource operator's output schema should contain columns that are needed by its parent and used by the filter operator. These columns which are only used by the filter operator, can be pruned.
For MPP mode, add a new projection above the DataSource operator is enough.
For Tikv mode, we also need to [push down the projection](https://github.com/pingcap/tidb/issues/51876) to tikv to reduce network overhead.
3. MPP physical join operator abandons the column prune achievements
During the process of building MPP tasks, physical join operaotr's schema is reset to its full semantic output:
https://github.com/pingcap/tidb/blob/b96f08134d66196ca760a6caf4d0e9cd044fff97/pkg/planner/core/task.go#L521
We'd better keep the column prune achievements, so that MPP Join can improve performance further(https://github.com/pingcap/tiflash/issues/8296).
Contributor guide
Assessment
This issue has not been assessed yet.