nextcloud / nextcloud/integration_onedrive

A single interrupted batch ends the files import for good, while the UI keeps showing "importing"

Open
#138 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
JavaScript
Stars
41
Forks
12
Avg merge
6d 16h
Merged PRs (30d)
4

Description

How to use GitHub
  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Steps to reproduce
  1. Connect a OneDrive account and start a files import large enough to need many batches.
  2. While a batch is running, interrupt the PHP process — restart the cron service, reboot, deploy, kill a wedged download, or let it be OOM-killed. In my case a routine service restart during a system upgrade was enough: the unit stops cron.php with SIGTERM while the import job is running.
  3. Check the job list:
$ occ background-job:list -c 'OCA\Onedrive\BackgroundJob\ImportOnedriveJob'
(no output)

Nothing ever resumes, and the Data migration page still shows the import as in progress.

Expected behaviour

An interrupted batch is retried on a later cron run and the import continues on its own. If it truly cannot continue, the UI should stop reporting an import in progress, so the user can restart it from the interface.

Actual behaviour

The import stops permanently and silently. The UI keeps claiming it is running, so there is no indication anything is wrong, and the user cannot restart it from the interface either.

The cause is a gap between two points in the job lifecycle:

  • QueuedJob::start() (server) removes the job's oc_jobs row before run() executes.
  • OnedriveStorageAPIService::importOnedriveJob() re-adds the job with $this->jobList->add(ImportOnedriveJob::class, ['user_id' => $userId]) only at the end of a successful batch.

Any exit between those two points — signal, fatal, OOM — leaves no job row at all, while the importing_onedrive user value is still '1'.

Recovery is not possible from the UI, because startImportOnedrive() returns early while the flag is set:

$alreadyImporting = $this->config->getUserValue($userId, Application::APP_ID, 'importing_onedrive', '0') === '1';
if ($alreadyImporting) {
    return ['targetPath' => $targetPath];
}

so the import button silently does nothing. The user has to clear the flag out of band first, e.g.

occ user:setting <uid> integration_onedrive importing_onedrive 0

and only then start the import again.

The app already anticipates stale runs — importOnedriveJob() checks onedrive_import_running against Application::IMPORT_JOB_TIMEOUT — but that guard only helps if a job still exists to run it, which after an interruption it does not.

Secondary, same code path: startImportOnedrive() resets imported_size and last_onedrive_import_timestamp and deletes import_tree, but does not reset nb_imported_files. After a restart the file counter carries on from its old value while the byte counter restarts at zero, so the UI shows something like "2195 files imported (609.2 MiB)" when only 2 of those files were downloaded in the current run. Both counters should be reset together.

Note on restarting: because import_tree is deleted, a restart re-walks the whole drive over the Graph API. Files already present are correctly skipped by getFile() (nodeExists()return 0), so nothing is re-downloaded, but on a large drive the re-listing is slow and costs a request per folder.

Suggested fixes
  1. Re-add the job on failure — a finally, or catch-and-requeue — so an interrupted batch resumes on the next cron run.
  2. Or add a small TimedJob that re-queues ImportOnedriveJob whenever importing_onedrive === '1' and no job is queued for that user. This also covers the case where the process had to be killed.
  3. Reset nb_imported_files alongside imported_size in startImportOnedrive().
Server configuration

Web server: Nginx (behind Traefik)

Database: PostgreSQL 17.10

PHP version: 8.4.23

Nextcloud version: 34.0.2

integration_onedrive version: 3.5.2

List of activated apps
Enabled:
  - activity: 7.0.0
  - app_api: 34.0.0
  - appstore: 1.0.0
  - bruteforcesettings: 7.0.0
  - calendar: 6.5.2
  - circles: 34.0.0
  - cloud_federation_api: 1.18.0
  - comments: 1.24.0
  - contacts: 8.7.5
  - contactsinteraction: 1.15.0
  - dashboard: 7.14.0
  - dav: 1.39.0
  - deck: 1.18.3
  - federatedfilesharing: 1.24.0
  - federation: 1.24.0
  - files: 2.6.0
  - files_downloadlimit: 5.2.0
  - files_lock: 34.0.1
  - files_pdfviewer: 7.0.0-dev.0
  - files_reminders: 1.7.0
  - files_sharing: 1.26.0
  - files_trashbin: 1.24.0
  - files_versions: 1.27.0
  - firstrunwizard: 7.0.0-dev.0
  - forms: 5.3.5
  - groupfolders: 22.0.6
  - integration_giphy: 2.2.1
  - integration_onedrive: 3.5.2
  - logreader: 7.0.0
  - lookup_server_connector: 1.22.0
  - mail: 5.10.10
  - maps: 1.7.1
  - music: 3.1.1
  - nextcloud_announcements: 6.0.0
  - notes: 6.0.1
  - notifications: 7.0.0-dev.1
  - notify_push: 1.3.3
  - oauth2: 1.22.0
  - office: 1.0.0
  - password_policy: 6.0.0-dev.0
  - photos: 7.0.0
  - privacy: 6.0.0-dev.1
  - profile: 1.3.0
  - provisioning_api: 1.24.0
  - recommendations: 7.0.0
  - related_resources: 5.0.0-dev.0
  - serverinfo: 6.0.0
  - settings: 1.17.0
  - sharebymail: 1.24.0
  - spreed: 24.0.3
  - support: 6.0.0
  - survey_client: 6.0.0-dev.0
  - systemtags: 1.24.0
  - tasks: 0.18.1
  - text: 8.0.0
  - theming: 2.9.0
  - twofactor_backupcodes: 1.23.0
  - twofactor_totp: 16.0.0
  - twofactor_webauthn: 2.7.0
  - updatenotification: 1.24.0
  - user_status: 1.14.0
  - viewer: 7.0.0-dev.0
  - weather_status: 1.14.0
  - webhook_listeners: 1.6.0
  - workflowengine: 2.16.0
Disabled:
  - admin_audit: 1.24.0
  - encryption: 2.22.0
  - files_external: 1.26.0
  - onlyoffice: 10.1.2 (installed 10.1.2)
  - suspicious_login: 12.0.0-dev.0
  - twofactor_nextcloud_notification: 8.0.0
  - user_ldap: 1.25.0
Nextcloud configuration ```json { "system": { "apps_paths": [ { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/apps", "url": "\/apps", "writable": false }, { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/nix-apps", "url": "\/nix-apps", "writable": false }, { "path": "\/nix\/store\/pppzqxmrmg6rjx387c2sp1cmqzmd00gx-nextcloud-34.0.2-with-apps\/store-apps", "url": "\/store-apps", "writable": true } ], "memcache.local": "\\OC\\Memcache\\APCu", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbtype": "pgsql", "datadirectory": "***REMOVED SENSITIVE VALUE***", "default_certificates_bundle_path": "\/nix\/store\/30wqfg7d7nxlqmlnxad14s2r775bq5mw-nss-cacert-3.125\/etc\/ssl\/certs\/ca-bundle.crt", "default_phone_region": "BR", "enabledPreviewProviders": [ "OC\\Preview\\PDF", "OC\\Preview\\SVG", "OC\\Preview\\BMP", "OC\\Preview\\GIF", "OC\\Preview\\JPEG", "OC\\Preview\\Krita", "OC\\Preview\\MarkDown", "OC\\Preview\\MP3", "OC\\Preview\\OpenDocument", "OC\\Preview\\PNG", "OC\\Preview\\TXT", "OC\\Preview\\XBitmap", "OC\\Preview\\HEIC" ], "forwarded_for_headers": [ "HTTP_X_FORWARDED_FOR", "HTTP_CF_CONNECTING_IP" ], "integrity.check.disabled": true, "log_type": "file", "loglevel": 1, "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_send_plaintext_only": false, "mail_sendmailmode": "smtp", "mail_smtpauth": true, "mail_smtpdebug": false, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": 465, "mail_smtpsecure": "ssl", "mail_smtpstreamoptions": [], "mail_smtptimeout": 10, "mail_template_class": "\\OC\\Mail\\EMailTemplate", "maintenance_window_start": 5, "maxUploadSize": "512M", "memcache.distributed": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "overwrite.cli.url": "REDACTED", "overwriteprotocol": "", "profile.enabled": false, "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0 }, "skeletondirectory": "", "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "upgrade.disable-web": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "version": "34.0.2.1", "dbtableprefix": "oc_", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "twofactor_enforced": "true", "twofactor_enforced_groups": [], "twofactor_enforced_excluded_groups": [], "maintenance": false, "app_install_overwrite": [], "appstoreenabled": true, "allow_local_remote_servers": true, "logfile": "\/var\/log\/nextcloud\/nextcloud.log", "trusted_domains": [ "REDACTED", "REDACTED" ], "PASS": "***REMOVED SENSITIVE VALUE***" } } ```
Browser

N/A — server-side background job behaviour. The only UI symptom is the Data migration page permanently showing an import in progress, which follows from the server state described above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with QueuedJob::start(), OnedriveStorageAPIService::importOnedriveJob(), and startImportOnedrive(), following how ImportOnedriveJob is queued and how the importing_onedrive state is updated. Reproduce an interrupted batch if possible, then ensure the job can resume or the UI can restart it, and that imported_size and nb_imported_files are reset together.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.