labstack / labstack/onebox

ob status and execution list do not show an in-flight scheduled job, making an exclusive lock look orphaned

Open
#188 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3
Forks
0
Avg merge
2h 40m
Merged PRs (30d)
63

Description

What happens

An operation was refused with:

✗ ob: application scheduling rendezvous remained busy — wait for the current
     scheduled job or application operation to finish

while, at the same moment, onebox reported that nothing was running:

$ ob execution list
no durable executions recorded

$ ob status
schedule <weekly-job>  last run failed: timeout (exit 143), and nothing has run since ⚠
schedule <5min-job>    skipped 20 firings in a row: an application operation is taking its lock ⚠

Host inspection showed the truth: a scheduled job declaring deploy_lock: exclusive had been running for nearly eight hours and legitimately held the rendezvous.

PID 365044  started 10:00:11  elapsed 07:57:46  PPID 1
            /bin/sh /etc/systemd/system/ob-<app>-<job>.run
PID 982137  started 15:47:12  docker compose run --rm --no-deps ... <job>

both holding fd 8 on <base>/<app>/schedule.lock. The application lock file itself did not exist, so nothing was stuck — the lock was doing exactly its job.

The bug

An in-flight scheduled job is invisible to ob status and ob execution list.

  • ob execution list returns no durable executions recorded while a job has been running for hours.
  • ob status reports the previous run's outcome and adds "and nothing has run since", which is actively wrong and points away from the truth.

ob schedule list does show the current run's start time as LAST TRIGGER, so the information exists and is simply not joined up.

Why it matters

The refusal message is accurate but gives nothing to act on. Every diagnostic onebox offers then says the system is idle, so the natural conclusion is an orphaned lock from the earlier timeout — that is what I concluded, and it is wrong. The real answer, "a job with an exclusive lock has been running since 10:00 and will hold it until it finishes", is not reachable through any ob command. It took /proc/locks and an fd scan on the host to find it.

The confusion is compounded by skipped being the same word used for a healthy skip, so a schedule that has done nothing for hours looks unremarkable.

Suggested

  • Show in-flight scheduled jobs in ob status and ob execution list. A running exclusive job is the single most useful thing to know when an operation is refused.
  • Name the holder in the rendezvous remained busy error — job name and start time. The data is discoverable from /proc/locks plus an fd scan; surfacing it turns a dead end into a wait.
  • Do not report "nothing has run since" when something is running.
  • Consider whether repeated skipped outcomes deserve a distinct signal from ordinary ones.

Environment

  • Runner: ob v2026.9.3 (ffe61080c66b)
  • Host: Ubuntu, docker 29.1.3, compose 2.40.3
  • Jobs: one weekly with deploy_lock: exclusive and an 8h timeout, two with deploy_lock: pinned

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the implementations behind ob status, ob execution list, and ob schedule list, then trace how scheduled-job state and the exclusive rendezvous lock are recorded. Reproduce an in-flight exclusive job and compare each command's output; done means the running job, holder timing, and accurate status are visible without host-level lock inspection.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.