[Bug report] Local job executor marks jobs run by other servers as FAILED in multi-node deployments
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Version
main branch
### Describe what's wrong
When multiple Gravitino servers share the same metadata store and use the local job executor, jobs that exit successfully are intermittently recorded as `FAILED`.
`LocalJobExecutor` keeps job status only in the memory of the server that submitted the job. But every server runs `JobManager.pullAndUpdateJobStatus`, which lists all active jobs in the shared store and asks its own executor for their status. When server B polls a job submitted by server A, B's executor throws `NoSuchJobException`, and B marks the job as `FAILED`. The recorded status depends on which server polls first.
Related problems in the same setup:
1. Cancelling a job on a server that didn't submit it fails with a 500 error, because that server's executor can't find the job.
2. `LocalJobExecutor` runs only one job at a time regardless of `gravitino.jobExecutor.local.maxRunningJobs`. Its thread pool has a core size of 0 and an unbounded queue, so it never grows beyond one thread.
### Error message and/or stacktrace
The server that ran the job logs that it completed successfully. The incorrect `FAILED` status is written by another server, which logs:
```
Job job-xxx with execution id local-job-xxx under metalake xxx is not found in the external job executor, marking it as FAILED.
```
### How to reproduce
1. Deploy two Gravitino servers sharing the same relational metadata store, both using the local job executor.
2. Register a shell job template, e.g. `/bin/bash -c "echo ok"`, and run it several times through the load balancer.
3. Some successful jobs are recorded as `FAILED` after the status pull, while the server that ran them logs success.
4. Cancel a running job through a server that didn't submit it: the request fails with a 500 error.
### Additional context
_No response_
Contributor guide
Research direction
Start by tracing LocalJobExecutor and JobManager.pullAndUpdateJobStatus across the shared metadata-store flow, including status polling and cancellation. Check how executor ownership and maxRunningJobs are represented when multiple servers are involved. Done means successful jobs are not marked FAILED, cross-server cancellation works, and the configured concurrency limit is honored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100