apache / apache/datafusion-ballista
Graceful executor shutdown terminates active Flight streams
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Describe the bug**
When an executor receives SIGTERM or Ctrl-C while a client is fetching data through Arrow Flight, the executor exits before the active stream finishes. The client receives a stream or transport error and cannot finish reading the result.
This is only observable when no tasks are currently running. If a task is active, #2356 causes shutdown to hang before reaching Flight server shutdown, masking this issue.
**To Reproduce**
1. Start a scheduler and executor.
2. Run a query that produces a large result.
3. Wait for execution tasks to finish and begin consuming the result slowly through Flight.
4. Send SIGTERM or Ctrl-C to the executor while the Flight stream is active.
5. Observe that the stream terminates before all batches are received.
**Expected behavior**
The executor should stop accepting new Flight requests, allow active Flight streams to complete, and then exit.
**Additional context**
The Flight server uses tonic's `serve_with_shutdown`, which supports draining existing connections. However, its `JoinHandle` is not awaited after shutdown begins and is not included in `ShutdownNotifier` completion tracking.
As a result, `start_executor_process` can return while Flight is still streaming. The executor's Tokio runtime is then dropped, cancelling the active stream.
Relevant code:
- `ballista/executor/src/executor_process.rs`
- `flight_server_task`
- `ShutdownNotifier`
- `ballista/executor/src/flight_service.rs`
Contributor guide
Research direction
Start in ballista/executor/src/executor_process.rs and trace flight_server_task, ShutdownNotifier, and the Flight service in ballista/executor/src/flight_service.rs. Verify that shutdown waits for the Flight server task and active streams before start_executor_process returns, so the executor exits only after all batches are delivered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100