apache / apache/gravitino

[Improvement] LocalJobExecutor supports high concurrency.

Open
#9,253 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

Currently, there is only one thread pool and one bounded waiting queue (LinkedBlockingQueue), which cannot handle high-concurrency job requests.

### How should we improve?

1. First, consider one catalog and one thread pool;

2. Here, the jobStatus can be directly stored in the job mysql table, continuously retrieving jobs to be submitted from the table and submitting them to the thread pool.

Contributor guide

Open the contributing guide

Research direction

Start by reading the LocalJobExecutor entry point and the job MySQL table mentioned in the issue. Clarify how jobStatus is currently handled and determine the intended catalog, thread-pool, and job-retrieval design. Done should mean that high-concurrency job requests can be submitted without relying on one bounded waiting queue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.