Adding too many files locks the instance and requires a restart [PosixFS]
- Dominant language
- Go
- Stars
- 2.1k
- Forks
- 274
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 103
Description
## Describe the bug
Adding a directory with over 19,000 files to a user via file system copy results in the container pinning the CPU usage and requires a restart to resolve.
## Steps to reproduce
1. Create new ocis instance using PosixFS driver
2. Sign in as **admin**
3. Create user **test1**
4. Log in as user **test1**
5. Upload directory with 13 files successfully
6. File system copy 2 directories with 15 and 47 files successfully
7. Attempt to add directory with 19023 files in total which caused the host CPU to spike until the container was restarted
## Expected behavior
The container should process the files added via upload or file system copy without burning the CPU.
## Actual behavior
A number of errors were encountered as part of the initial configuration and testing. I will list each one with log entries.
1. Error: could not find space for path /var/lib/ocis-storage-users
This error is the first received, and occurs 86 times:
```console
ocis | 2025-04-07T03:39:58.296055180Z {"level":"error","service":"storage-users","pkg":"rgrpc","error":"could not find space for path /var/lib/ocis-storage-users","path":"/var/lib/ocis-storage-users","time":"2025-04-07T03:39:58Z","message":"could not assimilate item"}
```
Even though the error occurs it does not seem to affect anything. What does it mean, and what are the implications?
Additionally, some errors have the message "could not assimilate item" and others have the message "failed to assimilate item". Is there a difference between these?
2. Error: error happened in MultiHostReverseProxy
Shortly after the server was started, the web ui was accesses at `https://cloud.example.com` and the following errors appeared:
[20250407-ocis-01.log](https://gist.github.com/instantdreams/c820819ff6ae4cb2157bd82667233b97)
Accessing any of these directly result in correctly rendered asset, e.g. `https://cloud.example.com/icons/arrow-drop-left-line.svg`, so I am not sure what this error relates to.
3. Issues with spaces on initial login
Server was started at 2025-04-07T03:39:57. Logged in as user **admin** at 03:41:10 and the log indicates errors, but the user interface was working:
[20250407-ocis-02.log](https://gist.github.com/instantdreams/f90d6281771b18cc9d6098a02ae45890)
There seems to be no discernable impact to these error messages.
4. Issues with spaces after creating new user
User "test1" was created at 2025-04-07T03:41:10. Space `shared` created at 2025-04-07T03:42:56. User `test1` added to space `shared`. Logging on as user `test1` produced similar error messages:
[20250407-ocis-03.log](https://gist.github.com/instantdreams/34c4dfc3d311f5ec1b3a20a430332d14)
I am unable to determine if there is any missing functionality due to these errors.
5. Entries for user and space management appearing in log
A folder was uploaded for `user1`. A set of file was uploaded into space `shared`. The logs filled with entries for these items:
[20250407-ocis-04.log](https://gist.github.com/instantdreams/f9381a42dc975856b59b93c0ddebc32f)
I do not expect these messages to appear in the log with my log level set to `error`. I do recognise that they are likely from a different logging system than the error messages. It might be a good idea to channel them to a different log output, or to assign them a level of `info`.
6. Files added via file system also generate log level entries
I copied in a set of files to `test1` and noted that similar log entries appeared as ocis indexed the new directory:
[20250407-ocis-05.log](https://gist.github.com/instantdreams/e8b1acc2035c7ef76017f5d62cf6a4d7)
There are a smattering of the similar space related error messages in with these, but the files are visible.
7. Adding too many files locks the instance and requires a restart
I uploaded a directory with 13 files, which were processed with errors but are successfully visible.
I added using filesystem copy two directories with 15 files and 47 files, which were also processed with errors but visible.
I then added using filesystem copy a directory with subdirectories that had a total of 19023 files. This spiked the CPU usage for the container, filled the logs with `mlock` entries, and ultimately failed with a nats error. Here's an extract from the log:
[20250408-ocis-06.log](https://gist.github.com/instantdreams/2f2fa2832410bdc16a9cbb756a51caea)
Here's an overview of the CPU usage (compared against the `pydio` containers):

The first upload via the web ui resulted in the following CPU usage:

Adding a **space** and copying files into a user directory via the file system resulted in this CPU usage:

Adding a directory with just over 19,000 files caused the following:

The server was restarted using commands `docker compose down;sleep 2;docker compose up --detach` and the log files captured after 2 hours of running:
[20250408-ocis-07.log](https://gist.github.com/instantdreams/682474fa5ed0367acf103939c61fc3c7)
The CPU usage is also a little excessive as well:

I am going to let the container run for a few more hours to see if everything calms down, but this isn't really usable.
## Setup
I am hosting the ocis service using docker on an Intel NUC8i5BEH with 32GB RAM and 500GB SSD running Debian 12.9, Docker 28.0.0, and Docker Compose 2.33.0. The server is connected to a thunderbolt 8 bay drive array which is has 8 x 8TB drives in a ZFS striped pool.
I wish to use ocis to manage files in an existing folder structure for my users. I have exposed the OCIS web ui via port 9292 and can access the instance via url `https://cloud.example.com` using traefik and a wildcard certificate. I have configured the ocis container to use the PosixFS driver using the [PosixFS](https://doc.owncloud.com/ocis/7.1/deployment/storage/posixfs.html) storage specifications.
The following volumes are mounted:
Container | Host | Type
----------|----------|----------
/etc/ocis | /srv/ocis/config | ext4
/var/lib/ocis | /srv/ocis/config | ext4
/var/lib/ocis-thumbnails | /srv/ocis/thumbnails | ext4
/var/lib/ocis-storage-users | /mnt/storage/ocis | zfs
Here is my `compose.yaml` file:
```yaml
services:
ocis:
container_name: ${CONTAINER1_NAME}
image: owncloud/ocis:${CONTAINER1_VERSION}
hostname: ${CONTAINER1_HOSTNAME}
domainname: ${DOMAINNAME}
entrypoint:
- /bin/sh
command: ["-c", "ocis init || true; ocis server"]
ports:
- ${CONTAINER1_PORT1}:9200 # web ui
env_file:
- .env
- .env.secrets
volumes:
- ${DIRECTORY_CONFIG}:/etc/ocis
- ${DIRECTORY_DATA}:/var/lib/ocis
- ${DIRECTORY_THUMBNAILS}:/var/lib/ocis-thumbnails
- ${DIRECTORY_USERS}:/var/lib/ocis-storage-users
- /etc/timezone:/etc/timezone:ro
restart: unless-stopped
```
Here is my `.env` file:
```
# Host specifics
CONTAINER1_NAME=ocis
CONTAINER1_VERSION=latest
CONTAINER1_HOSTNAME=ocis
CONTAINER1_PORT1=9292
DOMAINNAME=example.com
# Timezone from https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TZ=America/Edmonton
# Directory locations
DIRECTORY_CONFIG=/srv/ocis/config
DIRECTORY_DATA=/srv/ocis/data
DIRECTORY_THUMBNAILS=/srv/ocis/thumbnails
DIRECTORY_USERS=/mnt/storage/ocis
# Container specifics
## docker specific
OCIS_DOCKER_TAG=latest
OCIS_LOG_LEVEL=error
OCIS_INSECURE=true
OCIS_URL=https://cloud.example.com
## proxy
PROXY_TLS=false
## thumbnails
THUMBNAILS_FILESYSTEMSTORAGE_ROOT=/var/lib/ocis-thumbnails
## idm
IDM_CREATE_DEMO_USERS=false
## notifications
NOTIFICATIONS_SMTP_HOST=smtp.sendgrid.net
NOTIFICATIONS_SMTP_PORT=465
NOTIFICATIONS_SMTP_SENDER=admin@example.com
NOTIFICATIONS_SMTP_INSECURE=false
NOTIFICATIONS_SMTP_AUTHENTICATION=auto
NOTIFICATIONS_SMTP_ENCRYPTION=starttls
## storage users
STORAGE_USERS_DRIVER=posix
STORAGE_USERS_ID_CACHE_STORE=nats-js-kv
STORAGE_USERS_ID_CACHE_STORE_NODES=127.0.0.1:9233
STORAGE_USERS_POSIX_ROOT=/var/lib/ocis-storage-users
STORAGE_USERS_POSIX_PERSONAL_SPACE_PATH_TEMPLATE=users/{{.User.Username}}
STORAGE_USERS_POSIX_GENERAL_SPACE_PATH_TEMPLATE=projects/{{.SpaceName}}
STORAGE_USERS_POSIX_SCAN_DEBOUNCE_DELAY=4s
STORAGE_USERS_POSIX_USE_SPACE_GROUPS=true
NOTIFICATIONS_SMTP_PASSWORD=[password]
```
Here is my `.env.secrets` file:
```
# Container secrets
IDM_ADMIN_PASSWORD=[password]
NOTIFICATIONS_SMTP_USERNAME=[username]
NOTIFICATIONS_SMTP_PASSWORD=[password]
```
The service was started using the command `docker compose pull;docker compose down;sleep 2;docker compose up --detach`. The logs were monitored to make sure any initialisation processes were completed before accessing the web ui and logging in as the **admin** account.
## Additional context
CPU usage spiking when processing added files.
Contributor guide
Assessment
This issue has not been assessed yet.