apache / apache/seatunnel

[Improve][Zeta] Budget promoted cooperative workers in TaskExecutionService

Open
#12,121 3 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

## Description

This is a focused task for worker-side thread budgeting in `TaskExecutionService`. It is not a regression report.

Verified at `dev` commit `97d461bc0773399d632fd078735736ecd44f5f0b`:

- `TaskExecutionService.java:169-170` unbounded `LinkedBlockingDeque threadShareTaskQueue`; `:173-174` `executorService = newCachedThreadPool(new BlockingTaskThreadFactory())`; `:177-178` `RunBusWorkSupplier(executorService, threadShareTaskQueue)`.
- The cooperative worker requeues unfinished tasks at the queue tail. `TaskCallTimer.timeoutAct` (`TaskCallTimer.java:130-143`) marks the current bus-work exclusive to a slow task and submits a new bus-work via `runBusWorkSupplier.runNewBusWork(false)`, so each slow cooperative call adds one more thread.

Slot count bounds the number of task groups, not the number of threads or queue entries. No claim is made here about one thread per source split; the growth path is admitted engine tasks, callbacks, and promoted cooperative workers.

## Expected outcome

- A per-slot and per-job budget for promoted cooperative workers, with metrics for active threads, queue depth, and promotions.
- Any hard cap must still let source, coordinator, and sink tasks start and reach readiness; blocking the queue arbitrarily is not safe backpressure.
- Test: deploy many slow cooperative tasks and assert a bounded thread ceiling with no readiness deadlock.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with TaskExecutionService.java:169-178 and TaskCallTimer.java:130-143, then trace RunBusWorkSupplier and the cooperative worker requeue path. Validate the design by deploying many slow cooperative tasks; done means bounded promoted-worker threads with active-thread, queue-depth, and promotion metrics, while source, coordinator, and sink tasks reach readiness without deadlock.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.