Ballista Enhancement Overview

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Current Ballista implementation is more like a POC product for verification of whether it's able to run the Datafusion operators in a distributed way. It helps set up the whole framework and works well for just verification. However, it's a long way to introduce it to the production environment for real cases. This issue mainly raises several aspects we need to consider and to enhance for a more robust distributed execution framework.

In big data era, there're many scenarios. Two common ones are query for interactive analysis and batch processing for ETL purpose. There's no silver bullet. Each scenario has its own characteristics and has its own needs. In the following, I'll describe some enhancement we can do for each scenario.

For both interactive query and batch processing:
- [ ] [Necessary] apache/arrow-ballista#6
- [x] [Necessary] apache/arrow-datafusion#1703
- [ ] [Necessary] apache/arrow-datafusion#1704
- [ ] [Necessary] Support to fast recovery of scheduler restarting
- [ ] [Necessary] Support to better handle executor lost
- [ ] [Necessary] Support to better manage configurations
- [ ] [Nice to have] Support to schedule stages based on priority
- [ ] [Nice to have] Support to cancel SQL or cancel Job
- [ ] [Nice to have] Support executor blacklist

For interactive query:
- [x] [Necessary] Support push-based task assignment apache/arrow-datafusion#1221
- [ ] [Necessary] Support better data exchange, don't spill to disk apache/arrow-datafusion#1805
- [ ] [Necessary] Support better result fetching, don't spill to disk

For batch processing:
- [ ] [Necessary] Support task speculative scheduling
- [ ] [Necessary] Support shuffle fetch failure handling and retry
- [ ] [Necessary] Support to reattempt some stages

No files, tests, or implementation entry points are named. Start by reviewing the Ballista and DataFusion items listed in the overview and select a single, separately scoped enhancement. Done would require a defined production scenario, implementation scope, and acceptance criteria for that one item.

Written by the indexing model from the issue text.