owncloud / owncloud/ocis

Recursive mlocks created in PosixFS

Open
#11,093 1 comment 1 reaction 0 assignees View on GitHub
Type:Bug
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

I've recently was able to spin up oCIS 7.0.1 (docker) with the new PosixFS. It's certainly an excellent addition overall. I tried to stress-test it by uploading thousands of files. While overall it handled them, it seems that creation of a new file in the parent directory (personal space) of the user triggers creation and deletion of `.mlock`s for ALL files, i.e. in this directory as well as in all child directories recursively. On the other hand, creation of a new folder in the parent directory only triggers creation and deletion of `.mlock`s in this directory itself.
I am not sure if this is a bug or whether this may belong better to the https://github.com/cs3org/reva repository, but it clearly is a very problematic behavior which leads oCIS to make the server busy for several minutes if you have thousands of files.

## Steps to reproduce
Consider the following directory structure, with folders A and B containing files A1.txt and B1.txt, respectively.

![Image](https://github.com/user-attachments/assets/09ce0fe6-a853-46d6-8766-1b763d7a3e9d)

Now create a new text file in the parent directory.
EDIT: It's worth noting that the same thing happens if you create files with web interface, or directly in the filesystem.

## Expected behavior
The creation of `.mlock`s in the same folder is probably expected. The creation of `.mlock` for ALL files seems excessive.

## Actual behavior
In actuality, there is the following "avalanche" of locks created:
```
2025/03/04 08:47:27 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:27 /data/users/username/,CREATE,New file.txt
2025/03/04 08:47:27 /data/users/,CREATE,username.mlock
2025/03/04 08:47:27 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:27 /data/users/,DELETE,username.mlock
2025/03/04 08:47:27 /data/users/,CREATE,username.mlock
2025/03/04 08:47:27 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:27 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:27 /data/users/username/,DELETE,New file.txt.mlock
2025-03-04T08:47:27Z ERR resource info is missing checksum service=thumbnails
2025/03/04 08:47:28 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/,CREATE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,.Trash.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",.Trash.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,.Trash.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,CREATE,files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,"CLOSE_WRITE,CLOSE",files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,DELETE,files.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,CREATE,info.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,"CLOSE_WRITE,CLOSE",info.mlock
2025/03/04 08:47:28 /data/users/username/.Trash/,DELETE,info.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,001.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,A.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",A.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,A.mlock
2025/03/04 08:47:28 /data/users/username/A/,CREATE,A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/A/,"CLOSE_WRITE,CLOSE",A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/A/,DELETE,A1.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,B.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",B.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,B.mlock
2025/03/04 08:47:28 /data/users/username/B/,CREATE,B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/B/,"CLOSE_WRITE,CLOSE",B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/B/,DELETE,B1.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,New file.txt.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,B.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",B.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,B.mlock
2025/03/04 08:47:28 /data/users/,CREATE,username.mlock
2025/03/04 08:47:28 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:28 /data/users/,DELETE,username.mlock
2025/03/04 08:47:28 /data/users/username/,CREATE,A.mlock
2025/03/04 08:47:28 /data/users/username/,"CLOSE_WRITE,CLOSE",A.mlock
2025/03/04 08:47:28 /data/users/username/,DELETE,A.mlock
2025/03/04 08:47:37 /data/users/username/,CREATE,New file.txt.mlock
2025/03/04 08:47:37 /data/users/,CREATE,username.mlock
2025/03/04 08:47:37 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:47:37 /data/users/username/,"CLOSE_WRITE,CLOSE",New file.txt.mlock
2025/03/04 08:47:37 /data/users/username/,DELETE,New file.txt.mlock
```
Now, if I create a New Folder in the parent directory, the behavior is more reasonable:
```
2025/03/04 08:50:21 /data/users/,DELETE,username.flock
2025/03/04 08:50:21 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:21 /data/users/username/,"CREATE,ISDIR",New folder
2025/03/04 08:50:21 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:21 /data/users/username/,DELETE,New folder.mlock
2025/03/04 08:50:21 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:21 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:50:22 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:22 /data/users/,"CLOSE_WRITE,CLOSE",username.mlock
2025/03/04 08:50:22 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:22 /data/users/username/,DELETE,New folder.mlock
2025/03/04 08:50:22 /data/users/username/,CREATE,New folder.mlock
2025/03/04 08:50:22 /data/users/username/,"CLOSE_WRITE,CLOSE",New folder.mlock
2025/03/04 08:50:22 /data/users/username/,DELETE,New folder.mlock
```
Similarly, if I create a file in the A dir, there are no locks in the B dir.
Overall, this behavior is a big problem if you have thousands of files.

## Setup
I use the docker-compose.yml presented below.
```
services:
ocis:
image: owncloud/ocis:7.0.1
user: "1000:1000"
environment:
# INSECURE: needed if oCIS / Traefik is using self generated certificates
OCIS_INSECURE: "true"

# OCIS_URL: the external domain / ip address of oCIS (with protocol, must always be https)
OCIS_URL: "https://ocis.example.com"
GATEWAY_GRPC_ADDR: "0.0.0.0:9142"
MICRO_REGISTRY_ADDRESS: "127.0.0.1:9233"
NATS_NATS_HOST: "0.0.0.0"
NATS_NATS_PORT: 9233

STORAGE_USERS_DRIVER: "posix"
STORAGE_USERS_POSIX_ROOT: "/data"
STORAGE_USERS_POSIX_WATCH_TYPE: "inotifywait"
STORAGE_USERS_ID_CACHE_STORE: "nats-js-kv"
STORAGE_USERS_ID_CACHE_STORE_NODES: "localhost:9233"
STORAGE_USERS_POSIX_USE_SPACE_GROUPS: "true"
# OCIS_LOG_LEVEL: error / info / ... / debug
OCIS_LOG_LEVEL: "error"
OCIS_LOG_COLOR: "true"
OCIS_LOG_PRETTY: "true"
PROXY_ENABLE_BASIC_AUTH: "false"

# OIDC with authentik
OCIS_OIDC_ISSUER: "https://auth.example.com/application/o/ocis/"
PROXY_OIDC_ISSUER: "https://auth.example.com/application/o/ocis/"
PROXY_USER_OIDC_CLAIM: "preferred_username"
WEB_OIDC_CLIENT_ID: "[redacted]"
PROXY_OIDC_REWRITE_WELLKNOWN: "true"
PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD: "none"
WEB_OIDC_SCOPE: "openid profile email groups"
PROXY_ROLE_ASSIGNMENT_DRIVER: "oidc"
PROXY_ROLE_ASSIGNMENT_OIDC_CLAIM: "ocisRoles"
PROXY_AUTOPROVISION_ACCOUNTS: "true"
PROXY_CSP_CONFIG_FILE_LOCATION: '/etc/ocis/csp.yaml'
OCIS_EXCLUDE_RUN_SERVICES: "idp"
GRAPH_ASSIGN_DEFAULT_USER_ROLE: "false"
GRAPH_USERNAME_MATCH: "none"
PROXY_USER_CS3_CLAIM: "username"
# SMTP
NOTIFICATIONS_SMTP_HOST: "smtp.example.com"
NOTIFICATIONS_SMTP_PORT: "465"
NOTIFICATIONS_SMTP_AUTHENTICATION: "plain"
NOTIFICATIONS_SMTP_SENDER: "OCIS notifications "
NOTIFICATIONS_SMTP_USERNAME: "ocis@example.com"
NOTIFICATIONS_SMTP_PASSWORD: "password"
NOTIFICATIONS_SMTP_ENCRYPTION: "ssltls"
OCIS_ADD_RUN_SERVICES: "notifications"
# SEARCH with tika
SEARCH_EXTRACTOR_TYPE: "tika"
SEARCH_EXTRACTOR_TIKA_TIKA_URL: "http://tika:9998"
FRONTEND_FULL_TEXT_SEARCH_ENABLED: "true"
SEARCH_ENGINE_BLEVE_DATA_PATH: "/var/lib/ocis/search"
SEARCH_ENGINE_TYPE: "bleve"

volumes:
# mount the ocis config file inside the container
- "./etc-ocis:/etc/ocis"
- "./ocis-data:/data"
- "./var-lib:/var/lib/ocis"
expose:
- 9200
labels:
- "traefik.enable=true"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}.rule=Host(`ocis.example.eu`)"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}.entrypoints=websecure"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}.tls.certresolver=tls-resolver"
- "traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.port=9200"
- "traefik.http.services.${COMPOSE_PROJECT_NAME}.loadbalancer.server.scheme=https"
# cors
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}.middlewares=ocis-cors"
- "traefik.http.middlewares.ocis-cors.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
- "traefik.http.middlewares.ocis-cors.headers.accesscontrolalloworiginlist=https://ocis.example.com,https://auth.example.com"
- "traefik.http.middlewares.ocis-cors.headers.accesscontrolmaxage=100"
- "traefik.http.middlewares.ocis-cors.headers.addvaryheader=true"
# MKCOL, MOVE body removal, workaround https://github.com/owncloud/ocis/issues/10809
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.rule=Host(`ocis.example.eu`) && (Method(`MKCOL`) || Method(`MOVE`))"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.entrypoints=websecure"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.tls.certresolver=tls-resolver"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.middlewares=ocis-mkcol-body"
- "traefik.http.routers.${COMPOSE_PROJECT_NAME}-mkcol.service=${COMPOSE_PROJECT_NAME}"
- "traefik.http.middlewares.ocis-mkcol-body.buffering.maxRequestBodyBytes=1"

tika:
image: apache/tika:3.1.0.0-full
# release notes: https://tika.apache.org
restart: always
logging:
driver: local

```

## Additional considerations
The presented log files come from the docker logs. A separate issue seems to be that I am not able to disable them. They are there if if I set log level to warn or error. Is this expected?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.