apache / apache/datafusion-ballista

Graceful executor shutdown terminates active Flight streams

Open
#2,357 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Describe the bug**

When an executor receives SIGTERM or Ctrl-C while a client is fetching data through Arrow Flight, the executor exits before the active stream finishes. The client receives a stream or transport error and cannot finish reading the result.

This is only observable when no tasks are currently running. If a task is active, #2356 causes shutdown to hang before reaching Flight server shutdown, masking this issue.

**To Reproduce**

1. Start a scheduler and executor.
2. Run a query that produces a large result.
3. Wait for execution tasks to finish and begin consuming the result slowly through Flight.
4. Send SIGTERM or Ctrl-C to the executor while the Flight stream is active.
5. Observe that the stream terminates before all batches are received.

**Expected behavior**

The executor should stop accepting new Flight requests, allow active Flight streams to complete, and then exit.

**Additional context**

The Flight server uses tonic's `serve_with_shutdown`, which supports draining existing connections. However, its `JoinHandle` is not awaited after shutdown begins and is not included in `ShutdownNotifier` completion tracking.

As a result, `start_executor_process` can return while Flight is still streaming. The executor's Tokio runtime is then dropped, cancelling the active stream.

Relevant code:

- `ballista/executor/src/executor_process.rs`
- `flight_server_task`
- `ShutdownNotifier`
- `ballista/executor/src/flight_service.rs`

Contributor guide

Open the contributing guide

Research direction

Start in ballista/executor/src/executor_process.rs and trace flight_server_task, ShutdownNotifier, and the Flight service in ballista/executor/src/flight_service.rs. Verify that shutdown waits for the Flight server task and active streams before start_executor_process returns, so the executor exits only after all batches are delivered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.