getsentry / getsentry/self-hosted

SeaweedFS reaches volume limit after upgrading to 26.8.0, despite free disk space

Open
#4,503 5 comments 0 reactions 0 assignees View on GitHub
Waiting for: Product Owner
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
15

Description

### Self-Hosted Version

26.8.0

### CPU Architecture

x86_64

### Docker Version

28.1.1

### Docker Compose Version

2.35.1

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade from 26.7.2 to 26.8.0

### Steps to Reproduce

1. Run Sentry self-hosted 26.7.2 with SENTRY_EVENT_RETENTION_DAYS=30.
2. Upgrade to Sentry self-hosted 26.8.0.
3. Upgrade completed successfully.
4. Generate new Sentry events.
5. New events do not appear in Sentry.
6. Check the SeaweedFS logs with:
`docker compose logs --tail=300 seaweedfs`
7. Repeated `No writable volumes and no free volumes left` errors.
8. Check the SeaweedFS volume status with:
`docker compose exec seaweedfs sh -c 'echo "volume.list" | weed shell'`
9. SeaweedFS reports `1156/1156` volumes in use with `free:0`.
10. Check the filesystem usage and approximately 23 GiB of disk space is still available.

### Expected Result

After upgrading to 26.8.0, Sentry should continue accepting and storing new events.

SeaweedFS should be able to create additional volumes when existing volumes cannot accept new data, provided there is sufficient disk space available.

### Actual Result

After upgrading from 26.7.2 to 26.8.0, new Sentry events stopped appearing.

The Sentry taskworker reported:
```text
taskworker.task.failed taskname='sentry.tasks.store.save_event_transaction'
namespace='ingest.transactions'
exception_type='ClientError'
exception_message='An error occurred (InternalError) when calling the PutObject operation: We encountered an internal error, please try again.'
```

SeaweedFS repeatedly reported:
```text
upload_chunked.go:276 chunked upload failed: assign volume: all filers failed, last error: assign volume: assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:nodestore replication:000 ttl: error: No writable volumes and no free volumes left ...

s3api_object_handlers_put.go:537 putToFiler: chunked upload failed: assign volume: all filers failed, last error: assign volume: assign volume: rpc error: code = Unknown desc = failed to find writable volumes for collection:nodestore replication:000 ttl: error: No writable volumes and no free volumes left ...
```

The filesystem was not full:
```text
Filesystem Size Used Available Use%
/dev/mapper/vgroot-lvroot
77.2G 50.9G 23.2G 69%
```

However, SeaweedFS reported:
```text
Topology volumeSizeLimit:1024 MB hdd(volume:1156/1156 active:1156 free:0 remote:0)
Disk hdd(volume:1156/1156 active:1156 free:0 remote:0)
```

The existing nodestore volumes were also significantly smaller than the 1 GiB volume size limit. For example:
```text
134.9M /data/nodestore_23.dat
135.0M /data/nodestore_29.dat
135.1M /data/nodestore_35.dat
135.1M /data/nodestore_5.dat
135.1M /data/nodestore_9.dat
...
137.2M /data/nodestore_28.dat
237.9M /data/nodestore_2.dat
```

Eventually rolled back the installation to 26.7.2 using filesystem snapshot, I couldnt have more downtime.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the SeaweedFS output from `docker compose logs --tail=300 seaweedfs` and the `volume.list` command shown in the report, then compare the self-hosted 26.7.2 and 26.8.0 configurations. Reproduce the upgrade with the stated retention setting and verify that new events are accepted and SeaweedFS can create writable volumes while disk space remains available.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, shell
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.