getsentry / getsentry/self-hosted
SeaweedFS reaches volume limit after upgrading to 26.8.0, despite free disk space
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 15
Description
### Self-Hosted Version
26.8.0
### CPU Architecture
x86_64
### Docker Version
28.1.1
### Docker Compose Version
2.35.1
### Machine Specification
- [x] My system meets the minimum system requirements of Sentry
### Installation Type
Upgrade from 26.7.2 to 26.8.0
### Steps to Reproduce
1. Run Sentry self-hosted 26.7.2 with SENTRY_EVENT_RETENTION_DAYS=30.
2. Upgrade to Sentry self-hosted 26.8.0.
3. Upgrade completed successfully.
4. Generate new Sentry events.
5. New events do not appear in Sentry.
6. Check the SeaweedFS logs with:
`docker compose logs --tail=300 seaweedfs`
7. Repeated `No writable volumes and no free volumes left` errors.
8. Check the SeaweedFS volume status with:
`docker compose exec seaweedfs sh -c 'echo "volume.list" | weed shell'`
9. SeaweedFS reports `1156/1156` volumes in use with `free:0`.
10. Check the filesystem usage and approximately 23 GiB of disk space is still available.
### Expected Result
After upgrading to 26.8.0, Sentry should continue accepting and storing new events.
SeaweedFS should be able to create additional volumes when existing volumes cannot accept new data, provided there is sufficient disk space available.
### Actual Result
After upgrading from 26.7.2 to 26.8.0, new Sentry events stopped appearing.
The Sentry taskworker reported:
```text
taskworker.task.failed taskname='sentry.tasks.store.save_event_transaction'
namespace='ingest.transactions'
exception_type='ClientError'
exception_message='An error occurred (InternalError) when calling the PutObject operation: We encountered an internal error, please try again.'
```
SeaweedFS repeatedly reported:
```text
upload_chunked.go:276 chunked upload failed: assign volume: all filers failed, last error: assign volume: assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:nodestore replication:000 ttl: error: No writable volumes and no free volumes left ...
s3api_object_handlers_put.go:537 putToFiler: chunked upload failed: assign volume: all filers failed, last error: assign volume: assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:nodestore replication:000 ttl: error: No writable volumes and no free volumes left ...
```
The filesystem was not full:
```text
Filesystem Size Used Available Use%
/dev/mapper/vgroot-lvroot
77.2G 50.9G 23.2G 69%
```
However, SeaweedFS reported:
```text
Topology volumeSizeLimit:1024 MB hdd(volume:1156/1156 active:1156 free:0 remote:0)
Disk hdd(volume:1156/1156 active:1156 free:0 remote:0)
```
The existing nodestore volumes were also significantly smaller than the 1 GiB volume size limit. For example:
```text
134.9M /data/nodestore_23.dat
135.0M /data/nodestore_29.dat
135.1M /data/nodestore_35.dat
135.1M /data/nodestore_5.dat
135.1M /data/nodestore_9.dat
...
137.2M /data/nodestore_28.dat
237.9M /data/nodestore_2.dat
```
Eventually rolled back the installation to 26.7.2 using filesystem snapshot, I couldnt have more downtime.
### Event ID
_No response_
Contributor guide
Research direction
Start with the SeaweedFS output from `docker compose logs --tail=300 seaweedfs` and the `volume.list` command shown in the report, then compare the self-hosted 26.7.2 and 26.8.0 configurations. Reproduce the upgrade with the stated retention setting and verify that new events are accepted and SeaweedFS can create writable volumes while disk space remains available.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, shell
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100