Reduce external storage latency with persistent backend connections

[!TIP]

Help move this idea forward

• Use the 👍 reaction to show support for this feature.
• Avoid commenting unless you have relevant information to add; unnecessary comments create noise for subscribers.
• Subscribe to receive notifications about status changes and new comments.

---

Is your feature request related to a problem? Please describe.

File share operations with external shares (particularly with small files) (done through WebDAV or, in many cases, even through the Web UI) are more than twice as slow than with the default local storage.

While some slowdown is expected, I've successfully traced the large majority of the slowdown to the fact that, because each PHP process is independent, for every file operation that gets spun up, PHP must use smbclient to open a connection to the backend server, perform the file operation, then tear the connection down.

Needless to say, this is a huge cost we pay in terms of latency.

If Nextcloud could have an "external files daemon" (perhaps hosted by an AppAPI microservice?) then the daemon could keep the backend connection open (at least for a while) and serve requests initiated by PHP much faster. Such an architecture also enables the execution of very long single / large batches of file operations (by making operations a submit/notify for completion pair) without fear that either the PHP process will get killed or the browser HTTP request will time out.

Performance is a feature. If my people prefer to (and have gotten used to) bypass Nextcloud for file operations because doing them on the file server is literal orders of magnitude much faster, I cannot fault them.

---

I explored an alternative workaround; in my case it was to have the Linux server running Nextcloud mount -t cifs a file share via SMB into the Nextcloud user's file name space. That way, because the SMB mount is protocol 3.1.1, the Nextcloud server has full freshness into the view of whatever is mounted there.

The performance for Nextcloud-originated file operations (WebDAV or UI) such as copies and deletes more than doubled in this case. That stands to reason, since the kernel is tasked with keeping the SMB connection alive, Nextcloud spends no cycles doing that on every PHP process. Tantalizing!

Unfortunately, under this scenario, Nextcloud simply does not update the view of the folder, until a occ files:scan is run for the user whose files were updated in his mounts.

A file share being modified "behind the back of Nextcloud" is a common use case / scenario -- after all, file shares and mounts are common in productivity scenarios like businesses -- so this workaround is a no-go for us; we just can't keep hammering our file server with files:scan --all every minute or so. If it was possible for the server to notify the client mount about changes in the share, this wouldn't be a problem — but regrettably inotify does not currently work across SMB mounts, despite kernel change notification being supported in our current version of SAMBA.

Contrast this to the behavior of external_files which by default does update its view of every folder the user visits, when he clicks on the folder. Whether using the Nextcloud UI or a WebDAV client, Nextcloud appears to make no effort to even stat() the mountpoint to see if it has changed in the least. It seems like Nextcloud local storage is (as documented) truly meant to be managed exclusively by Nextcloud.

---

To give you a little context on how slow things are:

• Listing a folder with 66 very small files with ls on a mounted WebDAV share (gvfs) takes about 1.3 seconds. This is already very slow since the same folder directly listed in the Nextcloud server is barely single milliseconds.
• When the same folder with the same contents is exposed to Nextcloud via external_files SMB, doing the same ls on a mounted WebDAV share of that folder is 3 seconds.

The SAMBA server has 10Gbps connectivity to the Nextcloud server, and killer storage I/O.

There are arguments to be made — especially looking at the HTTP log — that the process is so inefficient that each file requires a PROPFIND to collect its info for the listing. But as you can see from the delta, the PROPFIND done on a file saved in a local Nextcloud server is more than twice as fast as the same PROPFIND done on a file that Nextcloud sees through external_files (SAMBA).

Now imagine how this problem is way, way worse with folders that contains a couple thousand files (rare, but not unheard of).

In case you might be wondering: this problem is just as bad whether the external file share is configured to Check for changes either Never or Once every direct access.

(Of course, if non-external-files overhead were to be cut substantially with regards to the PROPFIND operation taking at least 60 milliseconds per file, that would be a very welcome improvement too! But I suspect fixing this for real will largely be at odds with the single-process PHP nature of Nextcloud, which is... less than ideal in 2026.)

---

Edited to update: ten PROPFIND on the folder with the 66 files in local storage is about 3.4 seconds. Ten PROPFIND on a copy of that folder backed by external_files is about 4 seconds. That's a 600 ms overhead (60 milliseconds per request) just because the folder is externally stored.

This is all in equipment that has very good I/O and networking. APC cache is enabled. Redis caching is enabled. Redis file locking is enabled. I imagine people running NC on a budget NAS must be tearing out their hair.

---

Attaching configuration:

<?php
$CONFIG = array (
  'loglevel' => 1,
  'log_type' => 'file',
  'log_type_audit' => 'file',
  'datadirectory' => '/var/lib/nextcloud/data/',
  'updatechecker' => false,
  'check_for_working_htaccess' => false,
  'asset-pipeline.enabled' => false,
  'assetdirectory' => '/var/lib/nextcloud',
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/usr/share/nextcloud/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/lib/nextcloud/apps',
      'url' => '/apps-appstore',
      'writable' => true,
    ),
  ),
  'passwordsalt' => 'SALT',
  'secret' => 'SECRET',
  'trusted_domains' => 
  array (
    0 => 'cloud.example.com',
  ),
  'dbtype' => 'mysql',
  'version' => '32.0.3.2',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'PASGUORD',
  'installed' => true,
  'instanceid' => 'INSTANCEID',
  'htaccess.RewriteBase' => '/',
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_domain' => 'example.com',
  'mail_from_address' => 'nextcloud+noreply',
  'mail_smtphost' => 'mail.example.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'nextcloud',
  'mail_smtppassword' => 'PASGUORD',
  'maintenance' => false,
  'enabledPreviewProviders' => 
  array (
    0 => 'OC\\Preview\\BMP',
    1 => 'OC\\Preview\\GIF',
    2 => 'OC\\Preview\\JPEG',
    3 => 'OC\\Preview\\Krita',
    4 => 'OC\\Preview\\MarkDown',
    5 => 'OC\\Preview\\MP3',
    6 => 'OC\\Preview\\OpenDocument',
    7 => 'OC\\Preview\\PNG',
    8 => 'OC\\Preview\\TXT',
    9 => 'OC\\Preview\\XBitmap',
    10 => 'OC\\Preview\\Movie',
    11 => 'OC\\Preview\\PDF',
  ),
  'preview_max_x' => 1024,
  'preview_max_y' => 1024,
  'theme' => '',
  'trusted_proxies' => 
  array (
    0 => '10.250.2.7',
  ),
  'overwrite.cli.url' => 'https://cloud.example.com',
  'app_install_overwrite' => 
  array (
    0 => 'news',
    1 => 'caniupdate',
    2 => 'inventory',
    3 => 'files_markdown',
    4 => 'mindmap_app',
    5 => 'facerecognition',
    6 => 'files_scripts',
    7 => 'otpmanager',
    8 => 'files_mindmap',
  ),
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'default_phone_region' => 'US',
  'maintenance_window_start' => 4,
  'memories.db.triggers.fcu' => true,
  'memories.exiftool_no_local' => true,
  'memories.vod.path' => '/var/lib/nextcloud/apps/memories/bin-ext/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '127.0.0.1',
    'port' => 6379,
    'timeout' => 0.0,
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
);

The issue names no source files, tests, or entry points. Start by reproducing the reported external_files SMB latency with folder listings and PROPFIND operations, then trace the external-storage request path. Done means a reviewed design and implementation for persistent backend connections that reduces per-operation latency while preserving external-share freshness behavior.