[DISCUSSION] Should Ballista add support for streaming eager execution?

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Ballista currently implements a batch execution model directly modeled after Apache Spark. Query stages are executed to completion, with output written to shuffle files. The plan is optionally re-optimized based on statistics from completed query stages (AQE). This is a well proven design that scales massively for ETL and analytical workloads.

This issue is to discuss whether we should consider extending Ballista to also support a model that is closer to DataFusion's in-process execution model, where the whole plan is executed eagerly and results are streamed between stages, bypassing the shuffle files. This can lead to faster time to first results in some cases, and lower latency queries.

**Describe the solution you'd like**

**Describe alternatives you've considered**

**Additional context**

No files, tests, or entry points are named. Start by reviewing Ballista's batch execution, shuffle-file, and AQE model against DataFusion's eager streaming model described in the issue. Done would require an agreed design and scope for whether and how streaming eager execution should be supported.

Written by the indexing model from the issue text.