apache / apache/gravitino

[Bug report] Local job executor marks jobs run by other servers as FAILED in multi-node deployments

Closed
#13,146 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Version

main branch

### Describe what's wrong

When multiple Gravitino servers share the same metadata store and use the local job executor, jobs that exit successfully are intermittently recorded as `FAILED`.

`LocalJobExecutor` keeps job status only in the memory of the server that submitted the job. But every server runs `JobManager.pullAndUpdateJobStatus`, which lists all active jobs in the shared store and asks its own executor for their status. When server B polls a job submitted by server A, B's executor throws `NoSuchJobException`, and B marks the job as `FAILED`. The recorded status depends on which server polls first.

Related problems in the same setup:

1. Cancelling a job on a server that didn't submit it fails with a 500 error, because that server's executor can't find the job.
2. `LocalJobExecutor` runs only one job at a time regardless of `gravitino.jobExecutor.local.maxRunningJobs`. Its thread pool has a core size of 0 and an unbounded queue, so it never grows beyond one thread.

### Error message and/or stacktrace

The server that ran the job logs that it completed successfully. The incorrect `FAILED` status is written by another server, which logs:

```
Job job-xxx with execution id local-job-xxx under metalake xxx is not found in the external job executor, marking it as FAILED.
```

### How to reproduce

1. Deploy two Gravitino servers sharing the same relational metadata store, both using the local job executor.
2. Register a shell job template, e.g. `/bin/bash -c "echo ok"`, and run it several times through the load balancer.
3. Some successful jobs are recorded as `FAILED` after the status pull, while the server that ran them logs success.
4. Cancel a running job through a server that didn't submit it: the request fails with a 500 error.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by tracing LocalJobExecutor and JobManager.pullAndUpdateJobStatus across the shared metadata-store flow, including status polling and cancellation. Check how executor ownership and maxRunningJobs are represented when multiple servers are involved. Done means successful jobs are not marked FAILED, cross-server cancellation works, and the configured concurrency limit is honored.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.