riverqueue / riverqueue/river

River job stuck at running

Open
#1,258 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
5.7k
Forks
179
Avg merge
15h 43m
Merged PRs (30d)
13

Description

We recently noticed that a lot of our river job(around 5000) is stuck at a state of running. We have a global timeout of 1 hour and thought that the rescue is suppose to resolve this? These are across different job queues.

River job stats

SELECT
  COUNT(*) AS stuck_count,
  COUNT(DISTINCT kind) AS distinct_kinds,
  COUNT(DISTINCT queue) AS distinct_queues,
  MIN(created_at) AS oldest_created_at,
  MAX(created_at) AS newest_created_at,
  NOW() - MIN(created_at) AS oldest_age,
  NOW() - MAX(created_at) AS newest_age,
  AVG(NOW() - created_at) AS avg_age,
  PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM NOW() - created_at)) * INTERVAL '1 second' AS median_age,
  MIN(attempt) AS min_attempts,
  MAX(attempt) AS max_attempts,
  AVG(attempt)::numeric(10,2) AS avg_attempts,
  COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '1 hour') AS not_touched_in_1h,
  COUNT(*) FILTER (WHERE attempted_at < NOW() - INTERVAL '6 hours') AS not_touched_in_6h
FROM river_jobs.river_job
WHERE state = 'running'
  AND created_at < NOW() - INTERVAL '1 day';

[
	{
		"stuck_count": 5133,
		"distinct_kinds": 42,
		"distinct_queues": 7,
		"oldest_created_at": "2026-02-24 22:08:09.618834+00",
		"newest_created_at": "2026-05-25 19:10:35.474235+00",
		"oldest_age": "91 days 00:29:38.338464",
		"newest_age": "1 day 03:27:12.483063",
		"avg_age": "40 days 29:02:53.066961",
		"median_age": "1155:35:03.048939",
		"min_attempts": 1,
		"max_attempts": 4,
		"avg_attempts": 1.08,
		"not_touched_in_1h": 5133,
		"not_touched_in_6h": 5133
	}
]


River version

github.com/riverqueue/river v0.31.0
github.com/riverqueue/river/rivertype v0.31.0

Let me know if any other information is helpful

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining River v0.31.0's global timeout and rescue behavior, then reproduce or investigate the provided query showing old jobs still in the running state. Done means the cause of the stale records across queues is identified and a verified resolution or regression coverage is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.