Continuous 100% CPU / endless re-sync loop when sync folder is on exFAT (Linux, inotify watcher)
- Dominant language
- C++
- Stars
- 3.9k
- Forks
- 1k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 127
Description
## Summary
On Linux, when a sync folder lives on an exFAT (or vfat/ntfs) mount, the `nextcloud` client ends up in a permanent ~30s re-sync loop that pins one CPU core at 100%, even when no file was actually touched by the user. Every sync run is immediately followed by a storm of inotify change notifications for thousands of old, untouched files, which re-triggers another full sync, forever.
I traced the root cause through the source, wrote a minimal patch, and verified it fixes the issue on the actual affected machine (details below).
## Environment
- Client built from source (Qt6, CMake `Release` build) - reproduced on a `daily/34.0.1`-based checkout, but the underlying watcher code (`src/gui/folderwatcher_linux.cpp`, `src/gui/folder.cpp`) is unchanged on current `master`.
- Ubuntu, dual-boot with Windows.
- Sync folder mounted on exFAT, shared between Windows and Linux:
```
/dev/sda2 on /media//WSPOLNY type exfat (rw,relatime,uid=1000,gid=1000,fmask=0000,dmask=0000,allow_utime=0022,iocharset=utf8,errors=remount-ro)
```
exFAT is a hard requirement here - the volume is shared with Windows, so migrating to ext4 is not an option.
- Large, multi-year directory tree (photos/video, several years of subfolders, thousands of files).
## Symptoms (before the fix)
`strace -c -p -f` during one of the loop iterations showed the time dominated by `futex` (39%), `ppoll` (16%), `restart_syscall` (15%), `poll` (14%), alongside tens of thousands of `newfstatat`/`pwrite64`/`statx`/`access` calls and ~5k `chmod` calls in a span of a few seconds.
The client log (`nextcloud.gui.folderwatcher`, `folderwatcher.cpp:287`) showed, immediately after every completed sync run (`"Sync run took 31299 ms"`), a burst of hundreds/thousands of:
```
[ info nextcloud.gui.folderwatcher ... ]: change on path "......"
```
For a subset of these the client already recognized the notification as bogus (`"Ignoring spurious notification for file ..."`, see `Folder::slotWatchedPathChanged()` in `src/gui/folder.cpp`), but a large enough fraction was not filtered out, so a real full sync run was triggered, which then produced the next wave of notifications, forever.
## Root cause
1. `FolderWatcherPrivate::slotAddFolderRecursive()` in `src/gui/folderwatcher_linux.cpp` registers a recursive `inotify_add_watch()` on every directory unconditionally (`IN_CLOSE_WRITE | IN_ATTRIB | IN_MOVE | IN_CREATE | IN_DELETE | IN_DELETE_SELF | IN_MOVE_SELF | IN_UNMOUNT | IN_ONLYDIR`), with no awareness of the underlying file system. There is no equivalent on Linux of the Windows-only `FileSystem::fileSystemForPath()` (`src/common/filesystembase.cpp`, guarded by `#ifdef Q_OS_WIN`) that could be used to detect FAT-family/NTFS-on-FUSE mounts and behave differently.
2. The only spurious-notification guard that exists (`Folder::slotWatchedPathChanged()`, `src/gui/folder.cpp` around line 706) filters purely on `FileSystem::fileChanged()` (size + mtime comparison). On the exFAT kernel driver, mtime and/or attribute round-trips are apparently not fully stable across the discovery phase (the large `chmod`/`statx` volume in the strace suggests permission-normalization on every cycle, which itself re-triggers `IN_ATTRIB`), so this check does not reliably classify these as spurious, and a real sync gets scheduled again.
3. There is no throttle/cooldown in that path either - every notification that is not classified as spurious immediately calls `scheduleThisFolderSoon()`.
4. Notably, `src/common/syncjournaldb.cpp` (`defaultJournalMode()`) already has explicit, documented exFAT-awareness for the sqlite journal mode, but only on Windows/macOS - this exFAT-related class of issues has apparently been dealt with before for other subsystems, just never extended to the Linux file watcher.
I could not pin down the exact kernel-level mechanism (mtime rounding vs. the `fmask=0000/dmask=0000` mount options defeating the client's own permission reconciliation) that produces the unstable metadata - that would need tracing against the actual kernel exfat driver. Functionally, removing inotify from the loop for this class of file systems removes the symptom regardless of the precise mechanism.
## Fix
Patch attached in full at the bottom of this issue (happy to open as a PR instead if preferred). This went through a second, more skeptical review pass before being posted here, which caught a couple of real problems in the first draft, both now fixed in the attached diff:
- Extend `FileSystem::fileSystemForPath()` to Linux/macOS via `QStorageInfo::fileSystemType()` (already used elsewhere in the codebase, e.g. `syncengine.cpp`, `accountwizardcontroller.cpp` - no new dependency). Previously this function existed only on Windows.
- In `Folder::registerFolderWatcher()` (`src/gui/folder.cpp`), check the folder's file system type before creating the inotify-based `FolderWatcher`, **gated to `#if defined(Q_OS_LINUX)`**. For `vfat/exfat/msdos/ntfs/ntfs3/fuseblk`, skip the recursive inotify watcher entirely and instead start a plain `QTimer` (default 60s, overridable via `OWNCLOUD_POLL_INTERVAL_SEC`, capped at 24h) that periodically calls the existing `slotScheduleThisFolder()`, the same entry point already used for real watcher-triggered syncs. This intentionally does not reimplement a tree-walking polling watcher: the sync engine's own discovery phase already walks the tree and diffs mtime/size on every scheduled run, so the only thing missing for these file systems is a bounded-rate trigger. The `Q_OS_LINUX` gate matters: an earlier draft of this patch had no platform check at all, and since `Folder::registerFolderWatcher()` is shared, non-platform-specific code, that draft would have matched `"NTFS"` (returned by `GetVolumeInformationW` for essentially every Windows drive) and silently disabled the working `ReadDirectoryChangesW`-based watcher for most Windows users. Restricting the check to Linux avoids that entirely; this bug is specific to inotify plus the Linux exfat/vfat/ntfs3 kernel drivers, and there is no evidence Windows' or macOS' native watcher backends are affected.
- Add an `OWNCLOUD_FORCE_POLLING_WATCHER=1` environment variable for manual override, independent of auto-detection, matching the existing `OWNCLOUD_*` env-var convention used elsewhere in `libsync` (`OWNCLOUD_LAZYOPS`, `OWNCLOUD_BLACKLIST_TIME_MIN`, etc).
- Two follow-up null-checks were needed since `_folderWatcher` can now legitimately stay null for the lifetime of a `Folder`: `Folder::startSync()` (`_engine->setFilesystemPermissionsReliable(...)`) and `Folder::slotCapabilitiesChanged()` (both previously dereferenced `_folderWatcher` unconditionally). The `setFilesystemPermissionsReliable(...)` fallback defaults to `false` (not `true`) when there is no watcher to consult, since FAT/exFAT/NTFS are exactly the file systems where permission bits are typically not meaningfully settable - defaulting to `true` there would have been backwards and caused needless per-cycle `UPDATE_METADATA` work on precisely the file systems this patch targets.
Known limitations, flagging both in case there is a preferred way to handle them:
- `FolderWatcher` also carries the office file-lock-release notification signals (`filesLockReleased`/`lockedFilesFound`); folders using the polling fallback lose near-instant lock-release detection and only pick it up on the next poll interval. Given the alternative is a permanent 100%-CPU loop, this seemed like an acceptable trade-off.
- The file system type is only checked once, the first time `registerFolderWatcher()` succeeds for a given `Folder` instance, and the decision (real watcher vs. polling) is then permanent for that instance. This means a removable/external exFAT drive that isn't mounted yet at the moment the folder is first registered (e.g. not yet plugged in) could get the real inotify watcher attached against its mountpoint's parent file system, and keep it even after the exFAT volume mounts later - reproducing the original bug for that specific timing case. Re-evaluating on every call (rather than only when neither a watcher nor the polling timer exists yet) would close this gap, at the cost of a bit more complexity; happy to add that if wanted.
## Verification
Built and installed the patched client on the affected machine (real account, real exFAT-mounted library, thousands of files spanning 2016-2025) and compared `strace -c -p -f` snapshots before and after:
| Syscall | Before (steady-state loop) | After (patched, idle) |
|---|---|---|
| `statx` | 15330 | 6 |
| `pwrite64` | 122849 | 0 |
| `chmod` | 4987 | 0 |
| `newfstatat` | 71080 | 0 |
| `access` | 39521 | 0 |
After the initial (legitimate, one-time) full discovery of the library completed, CPU usage on the process settled and stayed near-idle over a 10+ minute observation window, with no recurrence of the ~30s re-sync cycle. A second `strace -e trace=%file` sample during the idle period returned zero file-related syscalls for the duration it ran.
## To reproduce (without the fix)
1. Mount an exFAT volume containing a large (multi-thousand file), multi-year directory tree.
2. Point a Nextcloud sync folder at it.
3. Let an initial sync complete.
4. Watch `nextcloud --background` CPU usage and the `nextcloud.gui.folderwatcher` debug log - it settles into a permanent ~30s cycle instead of going idle.
## Patch
full diff
```diff
diff --git a/src/common/filesystembase.cpp b/src/common/filesystembase.cpp
index 3d2164c..d253ad0 100644
--- a/src/common/filesystembase.cpp
+++ b/src/common/filesystembase.cpp
@@ -13,6 +13,7 @@
#include
#include
#include
+#include
#include
#include
@@ -573,9 +574,9 @@ bool FileSystem::isSymLink(const QString &filename, const QFileInfo &fileInfo)
return re;
}
-#ifdef Q_OS_WIN
QString FileSystem::fileSystemForPath(const QString &path)
{
+#ifdef Q_OS_WIN
// See also QStorageInfo (Qt >=5.4) and GetVolumeInformationByHandleW (>= Vista)
QString drive = path.left(2);
if (!drive.endsWith(QLatin1Char(':')))
@@ -593,8 +594,14 @@ QString FileSystem::fileSystemForPath(const QString &path)
return QString();
}
return QString::fromUtf16(reinterpret_cast(fileSystemBuffer));
-}
+#else
+ // QStorageInfo reads this from /proc/mounts on Linux (mtab equivalent
+ // on macOS/BSD); it reports the kernel-level fstype, e.g. "exfat",
+ // "vfat", "ntfs3", "fuseblk" for FUSE-mounted volumes, or "ext4".
+ const QStorageInfo storageInfo(path);
+ return QString::fromUtf8(storageInfo.fileSystemType());
#endif
+}
bool FileSystem::remove(const QString &fileName, QString *errorString)
{
diff --git a/src/common/filesystembase.h b/src/common/filesystembase.h
index eab4170..745c741 100644
--- a/src/common/filesystembase.h
+++ b/src/common/filesystembase.h
@@ -154,12 +154,13 @@ namespace FileSystem {
*/
QString OCSYNC_EXPORT joinPath(const QString &path, const QString &file);
-#ifdef Q_OS_WIN
/**
- * Returns the file system used at the given path.
+ * Returns the file system type used at the given path (e.g. "NTFS",
+ * "exfat", "ext4"), or an empty string if it could not be determined.
*/
- QString fileSystemForPath(const QString &path);
+ QString OCSYNC_EXPORT fileSystemForPath(const QString &path);
+#ifdef Q_OS_WIN
/*
* This function takes a path and converts it to a UNC representation of the
* string. That means that it prepends a \\?\ (unless already UNC) and converts
diff --git a/src/gui/folder.cpp b/src/gui/folder.cpp
index 074c87e..d1887de 100644
--- a/src/gui/folder.cpp
+++ b/src/gui/folder.cpp
@@ -121,6 +121,10 @@ Folder::Folder(const FolderDefinition &definition,
connect(&_scheduleSelfTimer, &QTimer::timeout,
this, &Folder::slotScheduleThisFolder);
+ _watcherPollingFallbackTimer.setSingleShot(false);
+ connect(&_watcherPollingFallbackTimer, &QTimer::timeout,
+ this, &Folder::slotScheduleThisFolder);
+
connect(ProgressDispatcher::instance(), &ProgressDispatcher::folderConflicts,
this, &Folder::slotFolderConflicts);
@@ -1222,7 +1226,13 @@ void Folder::startSync(const QStringList &pathList)
}
_engine->setIgnoreHiddenFiles(_definition.ignoreHiddenFiles);
- _engine->setFilesystemPermissionsReliable(_folderWatcher->canSetPermissions());
+ // _folderWatcher is null when the polling fallback is in use (see
+ // registerFolderWatcher()), i.e. only for FAT/exFAT/NTFS-family file
+ // systems on Linux. There's no watcher-based reliability test to consult
+ // in that case, but those file systems are exactly the ones where
+ // permission bits are typically not settable/meaningful, so default to
+ // "not reliable" rather than "reliable".
+ _engine->setFilesystemPermissionsReliable(_folderWatcher ? _folderWatcher->canSetPermissions() : false);
correctPlaceholderFiles();
@@ -1706,7 +1716,9 @@ void Folder::slotHydrationFailed(int errorCode, int statusCode, const QString &e
void Folder::slotCapabilitiesChanged()
{
- if (_accountState->account()->capabilities().filesLockAvailable()) {
+ // _folderWatcher is null when the polling fallback is in use (see
+ // registerFolderWatcher()); there's nothing to (re)connect in that case.
+ if (_folderWatcher && _accountState->account()->capabilities().filesLockAvailable()) {
connect(_folderWatcher.data(), &FolderWatcher::filesLockReleased, this, &Folder::slotFilesLockReleased, Qt::UniqueConnection);
connect(_folderWatcher.data(), &FolderWatcher::lockedFilesFound, this, &Folder::slotLockedFilesFound, Qt::UniqueConnection);
}
@@ -1737,13 +1749,56 @@ void Folder::setSaveBackwardsCompatible(bool save)
_saveBackwardsCompatible = save;
}
+#if defined(Q_OS_LINUX)
+// File systems whose Linux kernel drivers are known to report unstable
+// metadata (mtime/attribute) round-trips, which makes the recursive
+// inotify-based FolderWatcher see endless bogus changes and re-sync in a
+// tight loop (100% CPU) even though nothing was actually modified by the
+// user. This is specific to inotify on Linux: Windows' ReadDirectoryChangesW
+// and macOS' FSEvents backends are unaffected, and on Windows in particular
+// "ntfs" is the ordinary, reliable filesystem rather than a quirky one, so
+// this heuristic must not run there.
+static bool fileSystemNeedsPollingFallback(const QString &fileSystemType)
+{
+ static const QStringList unreliableTypes = {
+ QStringLiteral("vfat"), QStringLiteral("exfat"), QStringLiteral("msdos"),
+ QStringLiteral("ntfs"), QStringLiteral("ntfs3"), QStringLiteral("fuseblk")
+ };
+ for (const auto &type : unreliableTypes) {
+ if (fileSystemType.compare(type, Qt::CaseInsensitive) == 0)
+ return true;
+ }
+ return false;
+}
+#endif
+
void Folder::registerFolderWatcher()
{
if (_folderWatcher)
return;
+ if (_watcherPollingFallbackTimer.isActive())
+ return;
if (!QDir(path()).exists())
return;
+#if defined(Q_OS_LINUX)
+ static const auto forcePollingEnv = qEnvironmentVariableIntValue("OWNCLOUD_FORCE_POLLING_WATCHER") != 0;
+ const auto fileSystemType = FileSystem::fileSystemForPath(path());
+ if (forcePollingEnv || fileSystemNeedsPollingFallback(fileSystemType)) {
+ auto pollIntervalSec = qEnvironmentVariableIntValue("OWNCLOUD_POLL_INTERVAL_SEC");
+ if (pollIntervalSec <= 0)
+ pollIntervalSec = 60;
+ else if (pollIntervalSec > 86400)
+ pollIntervalSec = 86400; // 1 day cap, also keeps *1000 well within int range
+ qCInfo(lcFolder) << "Local file system for" << path() << "is" << fileSystemType
+ << "- using polling instead of the file watcher"
+ << "(interval:" << pollIntervalSec << "s)";
+ _watcherPollingFallbackTimer.setInterval(pollIntervalSec * 1000);
+ _watcherPollingFallbackTimer.start();
+ return;
+ }
+#endif
+
_folderWatcher.reset(new FolderWatcher(this));
connect(_folderWatcher.data(), &FolderWatcher::pathChanged,
this, [this](const QString &path) { slotWatchedPathChanged(path, Folder::ChangeReason::Other); });
@@ -1764,6 +1819,8 @@ void Folder::registerFolderWatcher()
void Folder::disconnectFolderWatcher()
{
+ _watcherPollingFallbackTimer.stop();
+
if (!_folderWatcher) {
return;
}
diff --git a/src/gui/folder.h b/src/gui/folder.h
index 94ce4ab..af81f43 100644
--- a/src/gui/folder.h
+++ b/src/gui/folder.h
@@ -643,6 +643,14 @@ private:
*/
QScopedPointer _folderWatcher;
+ /**
+ * Fallback for local directories whose file system does not report
+ * reliable change notifications (e.g. exFAT/FAT/NTFS mounts on Linux,
+ * see registerFolderWatcher()). Periodically schedules a sync instead
+ * of relying on _folderWatcher.
+ */
+ QTimer _watcherPollingFallbackTimer;
+
/**
* Keeps track of locally dirty files so we can skip local discovery sometimes.
*/
```
Contributor guide
Research direction
Start with src/gui/folderwatcher_linux.cpp, src/gui/folder.cpp, and src/common/filesystembase.cpp, then review the attached diff and the existing QStorageInfo usage in syncengine.cpp or accountwizardcontroller.cpp. Verify the Linux exFAT reproduction and confirm that the client uses bounded polling without recurring full syncs or sustained CPU and file-system activity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, linux
- Domain
- desktop, operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100