nextcloud / nextcloud/server

[Bug]: CleanupBackgroundJobsJob stuck in infinite loop due to integer overflow on oc_job_runs.duration

Open
#62,324 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage 34-feedback bug
Dominant language
PHP
Stars
36.9k
Forks
5.2k
Avg merge
2d 3h
Merged PRs (30d)
713

Description

⚠️ This issue respects the following points: ⚠️
Bug description

OC\Core\BackgroundJobs\CleanupBackgroundJobsJob::reapCrashedJobs() gets stuck
in a self-perpetuating loop when trying to mark a crashed job as finished
in oc_job_runs, because the duration and ram_peak_usage columns are
defined as integer (max ~2.14 billion) while the value being written
(likely milliseconds since the run started) grows unbounded for a job
that has been "reserved" for a long time. Every cron run it recalculates
an ever-larger duration, tries to UPDATE the row, and fails with an
integer overflow — so the row is never actually marked CRASHED, and the
loop repeats forever, roughly every ~45-75 minutes, filling the log with
errors indefinitely.

In my case this ran 290 times over ~3 days before I manually
intervened.

Steps to reproduce
  1. Have a background job get "stuck" in a reserved/running state for an
    extended period (in my case likely due to a PHP process being killed
    mid-execution, e.g. container restart).
  2. OC\Core\BackgroundJobs\CleanupBackgroundJobsJob runs on schedule via
    cron.php and calls reapCrashedJobs(), which computes the elapsed
    duration for the stuck job and calls JobRuns::finished().
  3. Once the computed duration (in ms) exceeds ~2.14 billion (~24.8 days,
    or potentially much sooner depending on what value is actually being
    stored — see note below), the UPDATE oc_job_runs SET duration = ...
    query fails with a Postgres integer overflow.
  4. Because the UPDATE fails, the row is never updated/removed, so the
    next cron run repeats the same failing calculation indefinitely.
Expected behavior

reapCrashedJobs() should successfully mark the stuck job as CRASHED,
regardless of how long it was stuck, and not get caught in a retry loop.

Nextcloud Server version

34

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.5

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Updated from a MINOR version (ex. 32.0.1 to 32.0.2)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?
  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other
Configuration report

List of activated Apps

Nextcloud Signing status
No errors have been found.
Nextcloud Logs
{"reqId":"Wf61ddTNazwUljHYsYr2","level":3,"time":"2026-07-16T04:51:10+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"/cron.php","scriptName":"/cron.php","message":"Error while running background job OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob (id: 105369284056305664, arguments: null)","userAgent":"--","version":"34.0.1.2","occ_command":["/var/www/html/cron.php"],"exception":{"Exception":"OC\\DB\\Exceptions\\DbalException","Message":"An exception occurred while executing a query: SQLSTATE[22003]: Numeric value out of range: 7 ERROR:  value \"2529580843\" is out of range for type integer\nCONTEXT:  unnamed portal parameter $2 = '...'","Code":7,"Trace":[{"file":"/var/www/html/lib/private/DB/ConnectionAdapter.php","line":78,"function":"wrap","class":"OC\\DB\\Exceptions\\DbalException","type":"::","args":[{"__class__":"Doctrine\\DBAL\\Exception\\DriverException"},"","UPDATE `*PREFIX*job_runs` SET `status` = :dcValue1, `duration` = :dcValue2, `ram_peak_usage` = :dcValue3 WHERE `run_id` = :dcValue4"]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":279,"function":"executeStatement","class":"OC\\DB\\ConnectionAdapter","type":"->","args":["UPDATE `*PREFIX*job_runs` SET `status` = :dcValue1, `duration` = :dcValue2, `ram_peak_usage` = :dcValue3 WHERE `run_id` = :dcValue4",{"dcValue1":3,"dcValue2":2529580843,"dcValue3":0,"dcValue4":96083097412030464},{"dcValue1":2,"dcValue2":2,"dcValue3":2,"dcValue4":2}]},{"file":"/var/www/html/lib/private/BackgroundJob/JobRuns.php","line":59,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/core/BackgroundJobs/CleanupBackgroundJobsJob.php","line":62,"function":"finished","class":"OC\\BackgroundJob\\JobRuns","type":"->","args":[96083097412030464,2529580843,0,{"__class__":"OCP\\BackgroundJob\\JobStatus","name":"CRASHED","value":3}]},{"file":"/var/www/html/core/BackgroundJobs/CleanupBackgroundJobsJob.php","line":39,"function":"reapCrashedJobs","class":"OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob","type":"->","args":[]},{"file":"/var/www/html/lib/public/BackgroundJob/Job.php","line":50,"function":"run","class":"OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob","type":"->","args":[null]},{"file":"/var/www/html/lib/public/BackgroundJob/TimedJob.php","line":86,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->","args":[{"__class__":"OC\\BackgroundJob\\JobList"}]},{"file":"/var/www/html/core/Service/CronService.php","line":204,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->","args":[{"__class__":"OC\\BackgroundJob\\JobList"}]},{"file":"/var/www/html/core/Service/CronService.php","line":107,"function":"runCli","class":"OC\\Core\\Service\\CronService","type":"->","args":["cron",null]},{"file":"/var/www/html/cron.php","line":52,"function":"run","class":"OC\\Core\\Service\\CronService","type":"->","args":[null]}]}}
Additional info

Setup note: Nextcloud runs in the official nextcloud:apache Docker
image (Apache + PHP inside the container), with Caddy in front as a
reverse proxy. This is unrelated to the bug itself (which is a database
schema issue), but noted for completeness.

The affected table schema (as of my installation):

                 Table "public.oc_job_runs"
     Column     |   Type   | Collation | Nullable | Default 
----------------+----------+-----------+----------+---------
 run_id         | bigint   |           | not null | 
 class_id       | bigint   |           | not null | 
 pid            | integer  |           | not null | 
 status         | smallint |           | not null | 
 duration       | integer  |           | not null | 0
 ram_peak_usage | integer  |           | not null | 0

Both run_id and class_id are already bigint, but duration and
ram_peak_usage are integer, which seems inconsistent given they're
both derived from measurements (time elapsed / memory used) that can
reasonably exceed the int4 range, especially for a job stuck for an
extended period.

Workaround applied locally (until an official fix/migration exists):

DELETE FROM oc_job_runs WHERE run_id = <stuck_run_id>;
ALTER TABLE oc_job_runs ALTER COLUMN duration TYPE bigint;
ALTER TABLE oc_job_runs ALTER COLUMN ram_peak_usage TYPE bigint;

This resolved the loop immediately and the error has not recurred since.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with core/BackgroundJobs/CleanupBackgroundJobsJob.php, especially reapCrashedJobs(), then read lib/private/BackgroundJob/JobRuns.php and inspect the oc_job_runs schema described in the issue. Trace how duration and ram_peak_usage are written and verify the stuck run can be marked CRASHED without PostgreSQL overflow errors or repeated cron failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
php, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.