Proposal for more efficient disk-based shuffle mechanism

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The current shuffle mechanism is too basic and produces too many small shuffle files.

Tasks to implement new version:

- [ ] https://github.com/apache/arrow-ballista/issues/332
- [ ] Implement a new version of `ShuffleWriterExec` that can work with a coalesced input of multiple parallel input partitions
- [ ] Changes to scheduler tracking of shuffle files

**Describe the solution you'd like**
See https://docs.google.com/document/d/16SIEoniAWKSFt8XKDLsOfRQ0sU--5E9_OIyE42Zj808/edit?usp=sharing

**Describe alternatives you've considered**

**Additional context**

Start by reading linked issue #332 and the referenced design document, then trace ShuffleWriterExec and the scheduler's shuffle-file tracking. Done means the new disk-based shuffle version handles coalesced input partitions, reduces small shuffle files, and includes the required scheduler tracking changes.

Written by the indexing model from the issue text.