Emptying trash leaves orphaned decomposedfs nodes behind for large/deep subtrees (git repos in particular)
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 274
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 103
Description
## Bug: Emptying trash leaves orphaned decomposedfs nodes behind for large/deep subtrees (git repos in particular)
### Environment
- oCIS version: `8.1.0` (image `owncloud/ocis:8.1.0@sha256:2c712f9056fd9beac2f4aa6b8fea2d784c9cbf4c9ea67f2778ee52bc66a952fd`)
- Storage driver: decomposedfs, `ocis` blobstore (not S3NG)
- Single-node deployment via Docker Compose, data on a POSIX filesystem (ZFS-backed, if relevant)
- Personal space storage, not a project space
### Summary
After permanently emptying a user's trash bin via the web UI, `ocis backup consistency` reports thousands of "symlink missing" (orphaned) nodes in that user's space. Investigating one such orphan showed it to be the root of a **fully-formed git repository** (`.git` with intact `objects/`, `refs/`, `hooks/`, `config`, etc. — all still correctly linked to each other via symlinks) that is completely disconnected from the live file tree. The top-level node's own `parentid` metadata field points to a node ID that does not exist on disk at all.
This strongly suggests emptying trash does not fully/recursively reclaim node metadata for large or deeply-nested subtrees — it appears to unlink/remove the top of the item (freeing the reported disk usage for blob content) but leaves descendant node metadata (and possibly blob content) orphaned on disk, invisible to the user, and never cleaned up by any subsequent process.
### Impact: this isn't just wasted disk space
We didn't go looking for this bug directly — we found it while debugging why a **nightly restic backup of this oCIS instance's data directory had gone from a normal runtime to 90-120 minutes**, despite the logical data involved being under 15GB. The root cause turned out to be a combination of a few things, but the dominant one was sheer **file count**: one personal space alone had **135,695 decomposedfs node files on disk**, for a user whose actual real, currently-visible content is only **~2,933 files**. Any backup tool, sync tool, or filesystem scan that has to `stat()`/traverse every file (restic, rsync, `find`, antivirus scanners, etc.) pays a real, ongoing performance cost for this orphaned data indefinitely — it's not a one-time disk-space nuisance, it's a permanent tax on every subsequent filesystem-level operation over that data, and it will keep growing every time a large/deep item is deleted and its trash is emptied. This seems like it could be a meaningful, silent performance issue for any self-hosted oCIS admin who has ever deleted something like a synced repo, a `node_modules` tree, or similar.
### Prior related issue
This looks related to (possibly a regression or edge-case of) #8473 "Disk space is not reclaimed from docker volume after files are deleted," which had the identical user-facing symptom (empty trash bin, disk usage unchanged) and was fixed via a Reva bump (cs3org/reva#4533) in oCIS v5.0.0. We're seeing the same class of symptom on 8.1.0, well after that fix. My best guess (unconfirmed) is that the original fix handles typical folder sizes correctly, but there may be a scale or depth limit that a large git object store (many thousands of small, deeply-sharded loose objects) exceeds, causing a partial/incomplete recursive delete rather than a full failure. This would be consistent with a separate report we found of `ocis revisions purge` timing out due to "extremely slow file globbing operations" on large directories — if trash purge uses similar globbing/recursion internally, the same scaling issue could apply.
### Steps to reproduce (best guess based on our investigation — we did not control the original event, so treat as a hypothesis to verify)
1. Sync or upload a git repository (with real history — i.e. a non-trivial `.git/objects/` directory containing at least hundreds of loose objects) into a personal space.
2. Delete it via the client/web UI (moves to trash).
3. Let it sit in trash for some time (may or may not be relevant).
4. Permanently empty the trash bin via the web UI.
5. Run `ocis backup consistency -p /storage/users` (oCIS stopped, offline).
6. Observe "symlink missing" entries. For each one, resolve its type via `ocis decomposedfs metadata --root /storage/users --node dump`. If `type=2` (directory), list its own directory contents (the symlinks to its children) and recurse — the children are fully intact, correctly cross-referenced, and represent real (if now-inaccessible) data.
### What we found, concretely
- One personal space: `ocis backup consistency` reported **1,880** "symlink missing" nodes, and no other inconsistency category (no orphaned/missing blobs, no missing nodes, no malformed metadata).
- Total node (`.mpk`) files under that space's `storage/users/spaces//nodes/`: **135,695**.
- The space's actual live/current file count (via the Graph/WebDAV API, a full breadth-first crawl from the space root): **~2,933 files, ~291 directories**.
- One specific orphaned node (chosen arbitrarily from the 1,880) was confirmed via `ocis decomposedfs metadata dump` to be a directory named `.git`, whose own `parentid` metadata field references a node ID that returns "no such file or directory" when looked up directly — i.e., a genuinely dangling backward-reference, not merely "no symlink from a live parent."
- Following that `.git` node's own directory listing (a normal decomposedfs directory node's children are POSIX symlinks to the children's real node paths) showed the complete, expected contents of a git repository: `HEAD`, `config`, `description`, `hooks`, `index`, `info`, `logs`, `objects`, `packed-refs`, `refs`, `COMMIT_EDITMSG`, `FETCH_HEAD`, `ORIG_HEAD` — all present and individually resolvable.
- Recursively following all symlinks from that single `.git` node (`find -L ... -type f | wc -l`) counted **509 files** still present on disk under that one disconnected subtree alone.
- `ocis backup consistency`'s "symlink missing" count therefore appears to significantly *undercount* the true scope of orphaned data — it flags the single disconnected root of each orphaned subtree, but does not report (and, per the command's source, does not attempt to walk) the descendants of that subtree. In our case the 1,880 reported orphans could plausibly correspond to a much larger number of actual orphaned filesystem objects once descendants are counted, though we have not exhaustively walked all 1,880 to get an exact total.
### Expected behavior
Emptying trash (or otherwise permanently deleting an item) should recursively and completely remove all decomposedfs node metadata and blob content belonging to that item, regardless of its size or nesting depth, leaving no orphaned data behind.
### Actual behavior
For at least some items (empirically, ones with an unusually large/deep internal structure, like a git repository's object store), only the top-level node's link appears to be severed; descendant node metadata remains fully intact and consistent with itself, just permanently disconnected and unreclaimed.
### Secondary observation (possibly worth a separate issue)
`ocis backup consistency` treats each disconnected subtree's root as a single "symlink missing" finding without walking or reporting on its descendants. This makes the reported inconsistency count a poor proxy for the actual amount of orphaned data/disk space involved — an admin relying on the reported count (as we initially did) would significantly underestimate the real impact. It might be worth having the tool optionally report descendant counts/sizes for each orphaned subtree root, or at minimum noting in the docs that the count reflects disconnection points, not total affected objects.
### Additional context
We could not find any documented or community-known procedure for manually reclaiming this kind of orphaned data (checked `ocis decomposedfs metadata` — inspection/manipulation only, no delete; `ocis migrate decomposedfs` — schema migrations only, not a GC tool; searched GitHub issues and the ownCloud Central forum). The closest community precedent (a forum thread about a 4TB drive at 98% full with only 1.5TB of real data, also showing "missing parent id" errors) was resolved by the affected admin abandoning the broken space and migrating data to a freshly created one, rather than any in-place repair — suggesting this is a known-hard problem without existing tooling, not something we're missing.
Happy to provide the full node metadata dumps, the `backup consistency` output, or any other diagnostics that would help — this was found via careful confirmation at each step (source-code review of `ocis/pkg/backup/backup.go` and `services/nats/pkg/command/`, direct `decomposedfs metadata dump` inspection, and manual symlink-chain walking) rather than assumption.
Contributor guide
Assessment
This issue has not been assessed yet.