[Epic]: Google Summer of Code 2025 Improving Spilling Execution

### Is your feature request related to a problem or challenge?

To support queries that exceed available memory, DataFusion must spill intermediate results to disk. As a continuation of the community effort on external query execution, this epic aims to improve the robustness of spilling execution and explore further performance optimizations.

This includes tracking which queries fail under specific memory limits, fixing bugs in external query execution, and addressing inefficiencies in the current implementation. An additional goal is to explore the feasibility of applying experimental optimizations proposed in academic papers, such as adaptive compression.

### Describe the solution you'd like
#### 1. Stabilize Larger-Than-Memory Queries
#### User Experience & Testing
- [x] enable `TrackConsumersPool` by default in `datafusion-cli`
- [ ] improve err msg to guide how to adjust parameters
- [ ] https://github.com/apache/datafusion/issues/15517
- [x] https://github.com/apache/datafusion/issues/16160
- [ ] migrate tests to `insta`

#### Sort
Related tracking issues: https://github.com/apache/datafusion/issues/16131, https://github.com/apache/datafusion/issues/16132
- [ ] https://github.com/apache/datafusion/issues/16042
- [ ] https://github.com/apache/datafusion/issues/14748
- [x] https://github.com/apache/datafusion/issues/14692
#### Aggregate
- [ ] Integrate ExternalSorter
#### Join
- [ ] https://github.com/apache/datafusion/issues/15760

#### 2. Optimize Spill File Format
- [x] https://github.com/apache/datafusion/issues/16130

#### 3. Docs & Blog
- [ ] https://github.com/apache/datafusion/issues/16177
### Describe alternatives you've considered

While spilling for window functions and CTEs is currently not a focus, they remain potential areas for improvement.

### Additional context

Related work:
- https://github.com/apache/datafusion/issues/15271
- https://github.com/apache/datafusion/issues/13123

Start by reviewing the unchecked checklist items and their linked issues, including 15517, 16042, 14748, 15760, and 16177. There is no single file, test, or entry point named in this epic; completion would require selecting and resolving a specific subtask, with the corresponding linked issue defining what done means.

Written by the indexing model from the issue text.