owncloud / owncloud/ocis

nats events can grow unlimited

Open
#10,319 1 comment 2 reactions 0 assignees View on GitHub
Type:Bug
Dominant language
Go
Stars
2.1k
Forks
274
Avg merge
2d 1h
Merged PRs (30d)
103

Description

## Describe the bug
The "main-queue" event stream can grow unlimited. Unless the messages are manually deleted, this could fill the FS at some point.
```
root@c61752751a7c:/# nats stream info main-queue
Information for Stream main-queue created 2024-10-16 08:57:37

Subjects: main-queue
Replicas: 1
Storage: File

Options:

Retention: Limits
Acknowledgments: true
Discard Policy: Old
Duplicate Window: 2m0s
Allows Msg Delete: true
Allows Purge: true
Allows Rollups: false

Limits:

Maximum Messages: unlimited
Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: unlimited
Maximum Message Size: unlimited
Maximum Consumers: unlimited

Cluster Information:

Name: ocis-cluster
Leader: NAYBADR4VT4NWUPFPKBRUUPDWIR3P722TNE75VRLIMWHIL35JLNFAJFZ

State:

Messages: 148
Bytes: 156 KiB
First Sequence: 1 @ 2024-10-16 09:00:34
Last Sequence: 148 @ 2024-10-16 12:43:59
Active Consumers: 11
Number of Subjects: 1
```

It's very likely that new nats clients trying to read the events will start from the last event (unless configured otherwise). However, it doesn't mean that the previous queued messages won't be kept.

## Steps to reproduce
Interacting with OCIS usually generates new events. Uploading files through the web seems to be the easiest way.
1.
2.
3.

## Expected behavior
Nats (or any other event broker) should have retention policies in place. Old message are expected to be either processed already (which means they could be manually removed), or they could have become irrelevant.
While having the limit on the number of messages might be tricky, messages older than 2 weeks (for example) should be irrelevant and could be deleted. Although the FS usage might still be unknown (depending on number of users and overall OCIS usage), it should remain stable.

## Actual behavior
If we assume that 1 user generates events up to 1MB of FS usage per day, the expected behaviour would be that, for a 1000 user installation, the FS usage would remain stable in around 15GB with the 15 days retention policy regardless of how much time OCIS has been running. Without such policy, we might reach 360GB in one year and over 1TB in 3 years.
Note that this is just for the events without taking account actual OCIS storage usage for the files that users are uploading.

## Setup
Please describe how you started the server and provide a list of relevant environment variables or configuration files.

```console
OCIS_XXX=somevalue
OCIS_YYY=somevalue
PROXY_XXX=somevalue
```

## Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.