nextcloud / nextcloud/maps

[Bug]: Cron job used more than 300 MB of ram after executing job OCA\Maps\BackgroundJob\AddPhotoJob

Open
#1,352 1 comment 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

0. Needs triage bug feature: 🖼 photos
Dominant language
JavaScript
Stars
598
Forks
113
No merged PRs in 30d

Description

⚠️ This issue respects the following points: ⚠️
Bug description

As mentioned in comment https://github.com/nextcloud/server/issues/45682#issuecomment-2486859429
I kept getting these errors in NC log after upgrade to NC 29.0.9 + Maps 1.4.0

Warning | cron | Cron job used more than 300 MB of ram after executing job OCA\Maps\BackgroundJob\AddPhotoJob (id: 622614, arguments: {"photoId":30578,"userId":"luka"}): 296.4 MB (before: 283.7 MB) | "19. stu 2024. 21:55:04"
Warning | cron | Cron job used more than 300 MB of ram after executing job OCA\Maps\BackgroundJob\AddPhotoJob (id: 600786, arguments: {"photoId":83819,"userId":"luka"}): 286.5 MB (before: 280.5 MB) | "19. stu 2024. 21:41:34"
Warning | cron | Cron job used more than 300 MB of ram after executing job OCA\Maps\BackgroundJob\AddPhotoJob (id: 571892, arguments: {"photoId":47767,"userId":"luka"}): 287.9 MB (before: 284.6 MB) | "19. stu 2024. 21:25:12"
Warning | cron | Cron job used more than 300 MB of ram after executing job OCA\Maps\BackgroundJob\AddPhotoJob (id: 561832, arguments: {"photoId":25830,"userId":"luka"}): 287.9 MB (before: 285 MB) | "19. stu 2024. 21:16:33"

As I was informed this is new feature of NC29 to monitor background jobs for memory usage. And it seems that AddPhotoJob in Maps is using more memory than expected.

I got around it by moving these messages from warning to debug log by editing line 200 in /var/www/nextcloud/cron.php which says $logger->warning($message, ['app' => 'cron']);, but I was asked if I could open issue here as well.

Steps to reproduce

Upgrade to NC29 with Maps, and wait for background job to start scanning photos to add them to map. Eventually it should trigger 300MB warning. Large photo library might be required.

Expected behavior

Job to use less memory and not trigger constant warnings.

Installation method

Manual installation

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install of the server?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Is this bug present after an update or on a fresh install of the app?

Updated from a minor version

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?
  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other
Configuration report
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.9.2",
        "overwrite.cli.url": "https:\/\/***REMOVED SENSITIVE VALUE***\/",
        "htaccess.RewriteBase": "\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "default_phone_region": "HR",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": "true",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "password": "***REMOVED SENSITIVE VALUE***",
            "timeout": 0.5
        },
        "enable_file_metadata": true,
        "enable_previews": true,
        "preview_concurrency_new": 4,
        "preview_max_x": 1440,
        "preview_max_y": 1440,
        "preview_ffmpeg_path": "\/usr\/bin\/ffmpeg",
        "enabledPreviewProviders": [
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\StarOffice",
            "OC\\Preview\\MSOffice2003",
            "OC\\Preview\\MSOffice2007",
            "OC\\Preview\\MSOfficeDoc",
            "OC\\Preview\\PDF",
            "OC\\Preview\\Postscript",
            "OC\\Preview\\Image",
            "OC\\Preview\\XBitmap",
            "OC\\Preview\\Photoshop",
            "OC\\Preview\\Illustrator",
            "OC\\Preview\\JPEG",
            "OC\\Preview\\PNG",
            "OC\\Preview\\BMP",
            "OC\\Preview\\GIF",
            "OC\\Preview\\SVG",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\Movie",
            "OC\\Preview\\MP4",
            "OC\\Preview\\M4V",
            "OC\\Preview\\AVI",
            "OC\\Preview\\MOV",
            "OC\\Preview\\QT",
            "OC\\Preview\\MKV",
            "OC\\Preview\\MTS",
            "OC\\Preview\\M2TS",
            "OC\\Preview\\TS",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\WMV",
            "OC\\Preview\\ASF",
            "OC\\Preview\\3GP",
            "OC\\Preview\\WEBM",
            "OC\\Preview\\VOB",
            "OC\\Preview\\MP3",
            "OC\\Preview\\EPS",
            "OC\\Preview\\ICO",
            "OC\\Preview\\PS",
            "OC\\Preview\\WMF",
            "OC\\Preview\\WEBP",
            "OC\\Preview\\FLV",
            "OC\\Preview\\RAW"
        ],
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "memories.vod.path": "\/var\/www\/nextcloud\/apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.exiftool": "\/var\/www\/nextcloud\/apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "maintenance": false,
        "memories.vod.disable": false,
        "memories.vod.vaapi": true,
        "memories.vod.vaapi.low_power": true,
        "loglevel": 2,
        "memories.db.triggers.fcu": true,
        "maintenance_window_start": 10,
        "log_rotate_size": 157286400,
        "memories.gis_type": 1
    }
}
List of activated Apps
Enabled:
  - activity: 2.21.1
  - audioplayer: 3.4.1
  - camerarawpreviews: 0.8.6
  - checksum: 1.2.5
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - cloud_py_api: 0.1.9
  - comments: 1.19.0
  - contacts: 6.0.1
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.3
  - federatedfilesharing: 1.19.0
  - files: 2.1.1
  - files_downloadlimit: 2.0.0
  - files_linkeditor: 1.1.21
  - files_pdfviewer: 2.10.0
  - files_photospheres: 1.29.1
  - files_readmemd: 3.0.2
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - files_zip: 2.1.0
  - firstrunwizard: 2.18.0
  - gpxpod: 5.0.18
  - groupfolders: 17.0.5
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - maps: 1.4.0
  - mediadc: 0.3.9
  - memories: 7.4.1
  - metadata: 0.21.0
  - music: 2.0.1
  - notes: 4.11.0
  - notifications: 2.17.0
  - oauth2: 1.17.1
  - password_policy: 1.19.0
  - phonetrack: 0.8.1
  - photos: 2.5.0
  - previewgenerator: 5.7.0
  - provisioning_api: 1.19.0
  - recognize: 7.1.0
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - side_menu: 3.13.1
  - systemtags: 1.19.0
  - tasks: 0.16.1
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - bruteforcesettings: 2.9.0
  - encryption: 2.17.0
  - federation: 1.19.0 (installed 1.16.0)
  - files_external: 1.21.0
  - nextcloud_announcements: 1.18.0 (installed 1.15.0)
  - privacy: 1.13.0 (installed 1.10.0)
  - recommendations: 2.1.0 (installed 1.5.0)
  - related_resources: 1.4.0 (installed 1.3.0)
  - sharebymail: 1.19.0 (installed 1.16.0)
  - support: 1.12.0 (installed 1.9.0)
  - survey_client: 1.17.0 (installed 1.14.0)
  - suspicious_login: 7.0.0
  - user_ldap: 1.20.0
  - user_status: 1.9.0 (installed 1.6.0)
Nextcloud Signing status
No errors have been found.
Nextcloud Logs
{"reqId":"G7XPPZqHMfmIfOhg4Xbr","level":2,"time":"2024-11-19T20:55:04+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Cron job used more than 300 MB of ram after executing job OCA\\Maps\\BackgroundJob\\AddPhotoJob (id: 622614, arguments: {\"photoId\":30578,\"userId\":\"luka\"}): 296.4 MB (before: 283.7 MB)","userAgent":"--","version":"29.0.9.2","data":{"app":"cron"},"id":"673d95c222672"}
{"reqId":"EZUYTP4Bni7A3ABBaA1l","level":2,"time":"2024-11-19T20:41:34+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Cron job used more than 300 MB of ram after executing job OCA\\Maps\\BackgroundJob\\AddPhotoJob (id: 600786, arguments: {\"photoId\":83819,\"userId\":\"luka\"}): 286.5 MB (before: 280.5 MB)","userAgent":"--","version":"29.0.9.2","data":{"app":"cron"},"id":"673d95c222677"}
{"reqId":"m8AZloar8nnKf96cyTT4","level":2,"time":"2024-11-19T20:25:12+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Cron job used more than 300 MB of ram after executing job OCA\\Maps\\BackgroundJob\\AddPhotoJob (id: 571892, arguments: {\"photoId\":47767,\"userId\":\"luka\"}): 287.9 MB (before: 284.6 MB)","userAgent":"--","version":"29.0.9.2","data":{"app":"cron"},"id":"673d95c22267b"}
{"reqId":"oF8LvUyhLUlohM0n2Y34","level":2,"time":"2024-11-19T20:16:33+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"Cron job used more than 300 MB of ram after executing job OCA\\Maps\\BackgroundJob\\AddPhotoJob (id: 561832, arguments: {\"photoId\":25830,\"userId\":\"luka\"}): 287.9 MB (before: 285 MB)","userAgent":"--","version":"29.0.9.2","data":{"app":"cron"},"id":"673d95c22267f"}
Browser Console
Additional info

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the OCA\Maps\BackgroundJob\AddPhotoJob entry point named in the report and reproduce it through cron with a photo library. Compare its memory use against the reported warnings; the work is done when photo-processing jobs no longer trigger the excessive-memory warnings without relying on the cron.php logging workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.