apache / apache/arrow

Acero's Execution Plan never finishes.

Open
#33,593 3 comments 0 reactions 1 assignee Claimed by @westonpace View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

We have observed that sometimes an execution plan with a small input never finishes (the future returned by the ExecPlan::finished() method is never marked as finished), even though the generator in the sink node is exhausted and has returned nullopt.

This issue seems to happen at random, the same plan with the same input sometimes works (the plan is marked finished) and sometimes it doesn't. Since the ExecPlanImpl destructor forces the executing thread to wait for the plan to finish (when the plan has not yet finished) we enter in a deadlock waiting for a plan that never finishes.

Since this has only happened with small inputs and not in a deterministic way, we believe the issue might be in the ExecPlan::StartProducing method.

Our hypothesis is that after the plan starts producing on each node, each node schedules their tasks and they are  immediately finished (due to the small input) and somehow the callback that marks the future finished_ finished is never executed.

 
```java

Status StartProducing() {
...
Future<> scheduler_finished = util::AsyncTaskScheduler::Make([this(util::AsyncTaskScheduler* async_scheduler) {
...
scheduler_finished.AddCallback([this](const Status& st) { finished_.MarkFinished(st);});
...
}
```

**Reporter**: [Pau Garcia Rodriguez](https://issues.apache.org/jira/browse/ARROW-18431)
**Assignee**: [Weston Pace](https://issues.apache.org/jira/browse/ARROW-18431) / @westonpace

**Note**: *This issue was originally created as [ARROW-18431](https://issues.apache.org/jira/browse/ARROW-18431). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.