apache / apache/texera

Replace fixed Thread.sleep calls in SyncExecutionResource with a bounded result-readiness poll

Open
#5,713 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
314
Forks
187
Avg merge
1d 21h
Merged PRs (30d)
214

Description

### Task Summary
The synchronous execution endpoint (`SyncExecutionResource::executeWorkflowSync`) uses two unconditional `Thread.sleep(500)` calls to wait for the result writer to flush before reading results, adding ~1s of latency to every run. The engine already commits the result writer (via the worker's blocking `writerThread.join()`) before an operator's `COMPLETED` state becomes observable, so the data is on disk by the time we read it. Replace both sleeps with a bounded readiness-poll that compares each target operator's committed result-storage count against its expected output count from stats, with early-exit and a short timeout cap.
### Task Type
- [x] Performance

Contributor guide

Open the contributing guide

Research direction

Start at SyncExecutionResource::executeWorkflowSync and trace how target operators expose committed result-storage counts and expected output counts from stats. Replace the two fixed waits with a bounded readiness poll that exits early when counts match and stops at a short timeout; verify that completed synchronous executions still return their results without the fixed latency.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
api, backend, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.