ob status and execution list do not show an in-flight scheduled job, making an exclusive lock look orphaned
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 2h 40m
- Merged PRs (30d)
- 63
Description
What happens
An operation was refused with:
✗ ob: application scheduling rendezvous remained busy — wait for the current
scheduled job or application operation to finish
while, at the same moment, onebox reported that nothing was running:
$ ob execution list
no durable executions recorded
$ ob status
schedule <weekly-job> last run failed: timeout (exit 143), and nothing has run since ⚠
schedule <5min-job> skipped 20 firings in a row: an application operation is taking its lock ⚠
Host inspection showed the truth: a scheduled job declaring deploy_lock: exclusive had been running for nearly eight hours and legitimately held the rendezvous.
PID 365044 started 10:00:11 elapsed 07:57:46 PPID 1
/bin/sh /etc/systemd/system/ob-<app>-<job>.run
PID 982137 started 15:47:12 docker compose run --rm --no-deps ... <job>
both holding fd 8 on <base>/<app>/schedule.lock. The application lock file itself did not exist, so nothing was stuck — the lock was doing exactly its job.
The bug
An in-flight scheduled job is invisible to ob status and ob execution list.
ob execution listreturnsno durable executions recordedwhile a job has been running for hours.ob statusreports the previous run's outcome and adds "and nothing has run since", which is actively wrong and points away from the truth.
ob schedule list does show the current run's start time as LAST TRIGGER, so the information exists and is simply not joined up.
Why it matters
The refusal message is accurate but gives nothing to act on. Every diagnostic onebox offers then says the system is idle, so the natural conclusion is an orphaned lock from the earlier timeout — that is what I concluded, and it is wrong. The real answer, "a job with an exclusive lock has been running since 10:00 and will hold it until it finishes", is not reachable through any ob command. It took /proc/locks and an fd scan on the host to find it.
The confusion is compounded by skipped being the same word used for a healthy skip, so a schedule that has done nothing for hours looks unremarkable.
Suggested
- Show in-flight scheduled jobs in
ob statusandob execution list. A running exclusive job is the single most useful thing to know when an operation is refused. - Name the holder in the
rendezvous remained busyerror — job name and start time. The data is discoverable from/proc/locksplus an fd scan; surfacing it turns a dead end into a wait. - Do not report "nothing has run since" when something is running.
- Consider whether repeated
skippedoutcomes deserve a distinct signal from ordinary ones.
Environment
- Runner:
ob v2026.9.3 (ffe61080c66b) - Host: Ubuntu, docker 29.1.3, compose 2.40.3
- Jobs: one weekly with
deploy_lock: exclusiveand an 8h timeout, two withdeploy_lock: pinned
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the implementations behind ob status, ob execution list, and ob schedule list, then trace how scheduled-job state and the exclusive rendezvous lock are recorded. Reproduce an in-flight exclusive job and compare each command's output; done means the running job, holder timing, and accurate status are visible without host-level lock inspection.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100