[Bug]: CleanupBackgroundJobsJob stuck in infinite loop due to integer overflow on oc_job_runs.duration
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 36.9k
- Forks
- 5.2k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 713
Description
⚠️ This issue respects the following points: ⚠️
- This is a bug, not a question or a configuration/webserver/proxy issue.
- This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
- Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
- I agree to follow Nextcloud's Code of Conduct.
Bug description
OC\Core\BackgroundJobs\CleanupBackgroundJobsJob::reapCrashedJobs() gets stuck
in a self-perpetuating loop when trying to mark a crashed job as finished
in oc_job_runs, because the duration and ram_peak_usage columns are
defined as integer (max ~2.14 billion) while the value being written
(likely milliseconds since the run started) grows unbounded for a job
that has been "reserved" for a long time. Every cron run it recalculates
an ever-larger duration, tries to UPDATE the row, and fails with an
integer overflow — so the row is never actually marked CRASHED, and the
loop repeats forever, roughly every ~45-75 minutes, filling the log with
errors indefinitely.
In my case this ran 290 times over ~3 days before I manually
intervened.
Steps to reproduce
- Have a background job get "stuck" in a reserved/running state for an
extended period (in my case likely due to a PHP process being killed
mid-execution, e.g. container restart). OC\Core\BackgroundJobs\CleanupBackgroundJobsJobruns on schedule via
cron.phpand callsreapCrashedJobs(), which computes the elapsed
duration for the stuck job and callsJobRuns::finished().- Once the computed duration (in ms) exceeds ~2.14 billion (~24.8 days,
or potentially much sooner depending on what value is actually being
stored — see note below), theUPDATE oc_job_runs SET duration = ...
query fails with a Postgres integer overflow. - Because the UPDATE fails, the row is never updated/removed, so the
next cron run repeats the same failing calculation indefinitely.
Expected behavior
reapCrashedJobs() should successfully mark the stuck job as CRASHED,
regardless of how long it was stuck, and not get caught in a retry loop.
Nextcloud Server version
34
Operating system
Debian/Ubuntu
PHP engine version
PHP 8.5
Web server
Apache (supported)
Database engine version
PostgreSQL
Is this bug present after an update or on a fresh install?
Updated from a MINOR version (ex. 32.0.1 to 32.0.2)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
What user-backends are you using?
- Default user-backend (database)
- LDAP/ Active Directory
- SSO - SAML
- Other
Configuration report
List of activated Apps
Nextcloud Signing status
No errors have been found.
Nextcloud Logs
{"reqId":"Wf61ddTNazwUljHYsYr2","level":3,"time":"2026-07-16T04:51:10+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"/cron.php","scriptName":"/cron.php","message":"Error while running background job OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob (id: 105369284056305664, arguments: null)","userAgent":"--","version":"34.0.1.2","occ_command":["/var/www/html/cron.php"],"exception":{"Exception":"OC\\DB\\Exceptions\\DbalException","Message":"An exception occurred while executing a query: SQLSTATE[22003]: Numeric value out of range: 7 ERROR: value \"2529580843\" is out of range for type integer\nCONTEXT: unnamed portal parameter $2 = '...'","Code":7,"Trace":[{"file":"/var/www/html/lib/private/DB/ConnectionAdapter.php","line":78,"function":"wrap","class":"OC\\DB\\Exceptions\\DbalException","type":"::","args":[{"__class__":"Doctrine\\DBAL\\Exception\\DriverException"},"","UPDATE `*PREFIX*job_runs` SET `status` = :dcValue1, `duration` = :dcValue2, `ram_peak_usage` = :dcValue3 WHERE `run_id` = :dcValue4"]},{"file":"/var/www/html/lib/private/DB/QueryBuilder/QueryBuilder.php","line":279,"function":"executeStatement","class":"OC\\DB\\ConnectionAdapter","type":"->","args":["UPDATE `*PREFIX*job_runs` SET `status` = :dcValue1, `duration` = :dcValue2, `ram_peak_usage` = :dcValue3 WHERE `run_id` = :dcValue4",{"dcValue1":3,"dcValue2":2529580843,"dcValue3":0,"dcValue4":96083097412030464},{"dcValue1":2,"dcValue2":2,"dcValue3":2,"dcValue4":2}]},{"file":"/var/www/html/lib/private/BackgroundJob/JobRuns.php","line":59,"function":"executeStatement","class":"OC\\DB\\QueryBuilder\\QueryBuilder","type":"->","args":[]},{"file":"/var/www/html/core/BackgroundJobs/CleanupBackgroundJobsJob.php","line":62,"function":"finished","class":"OC\\BackgroundJob\\JobRuns","type":"->","args":[96083097412030464,2529580843,0,{"__class__":"OCP\\BackgroundJob\\JobStatus","name":"CRASHED","value":3}]},{"file":"/var/www/html/core/BackgroundJobs/CleanupBackgroundJobsJob.php","line":39,"function":"reapCrashedJobs","class":"OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob","type":"->","args":[]},{"file":"/var/www/html/lib/public/BackgroundJob/Job.php","line":50,"function":"run","class":"OC\\Core\\BackgroundJobs\\CleanupBackgroundJobsJob","type":"->","args":[null]},{"file":"/var/www/html/lib/public/BackgroundJob/TimedJob.php","line":86,"function":"start","class":"OCP\\BackgroundJob\\Job","type":"->","args":[{"__class__":"OC\\BackgroundJob\\JobList"}]},{"file":"/var/www/html/core/Service/CronService.php","line":204,"function":"start","class":"OCP\\BackgroundJob\\TimedJob","type":"->","args":[{"__class__":"OC\\BackgroundJob\\JobList"}]},{"file":"/var/www/html/core/Service/CronService.php","line":107,"function":"runCli","class":"OC\\Core\\Service\\CronService","type":"->","args":["cron",null]},{"file":"/var/www/html/cron.php","line":52,"function":"run","class":"OC\\Core\\Service\\CronService","type":"->","args":[null]}]}}
Additional info
Setup note: Nextcloud runs in the official nextcloud:apache Docker
image (Apache + PHP inside the container), with Caddy in front as a
reverse proxy. This is unrelated to the bug itself (which is a database
schema issue), but noted for completeness.
The affected table schema (as of my installation):
Table "public.oc_job_runs"
Column | Type | Collation | Nullable | Default
----------------+----------+-----------+----------+---------
run_id | bigint | | not null |
class_id | bigint | | not null |
pid | integer | | not null |
status | smallint | | not null |
duration | integer | | not null | 0
ram_peak_usage | integer | | not null | 0
Both run_id and class_id are already bigint, but duration and
ram_peak_usage are integer, which seems inconsistent given they're
both derived from measurements (time elapsed / memory used) that can
reasonably exceed the int4 range, especially for a job stuck for an
extended period.
Workaround applied locally (until an official fix/migration exists):
DELETE FROM oc_job_runs WHERE run_id = <stuck_run_id>;
ALTER TABLE oc_job_runs ALTER COLUMN duration TYPE bigint;
ALTER TABLE oc_job_runs ALTER COLUMN ram_peak_usage TYPE bigint;
This resolved the loop immediately and the error has not recurred since.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with core/BackgroundJobs/CleanupBackgroundJobsJob.php, especially reapCrashedJobs(), then read lib/private/BackgroundJob/JobRuns.php and inspect the oc_job_runs schema described in the issue. Trace how duration and ram_peak_usage are written and verify the stuck run can be marked CRASHED without PostgreSQL overflow errors or repeated cron failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100