[Improvement] LocalJobExecutor supports high concurrency.
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
Currently, there is only one thread pool and one bounded waiting queue (LinkedBlockingQueue), which cannot handle high-concurrency job requests.
### How should we improve?
1. First, consider one catalog and one thread pool;
2. Here, the jobStatus can be directly stored in the job mysql table, continuously retrieving jobs to be submitted from the table and submitting them to the thread pool.
Contributor guide
Research direction
Start by reading the LocalJobExecutor entry point and the job MySQL table mentioned in the issue. Clarify how jobStatus is currently handled and determine the intended catalog, thread-pool, and job-retrieval design. Done should mean that high-concurrency job requests can be submitted without relying on one bounded waiting queue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- backend, databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100