getsentry / getsentry/self-hosted

Migrating Vroom data into Seaweed should not interrupt service uptime

Open
#4,263 18 comments 2 reactions 1 assignee Claimed by @aldy505 View on GitHub
Bug
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 3h
Merged PRs (30d)
12

Description

### Self-Hosted Version

25.5.1

### CPU Architecture

x86_64

### Docker Version

29.3.1

### Docker Compose Version

5.1.1

### Machine Specification

- [x] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade from 25.5.1 to 26.3.1

### Steps to Reproduce

1. run install.sh

### Expected Result

Sentry is successfully upgraded

### Actual Result

Upgrade fails. Last lines of output before the failure:

```
upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f' (14659 bytes in 0.0 seconds, 4.71 MB/s) [6097783 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' -> 's3://profiles/1/99/fffe61af14e64b9cb9f61900e3ced12f.attrs' (211 bytes in 0.0 seconds, 81.24 KB/s) [6097784 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53' (22012 bytes in 0.0 seconds, 6.63 MB/s) [6097785 of 6097786]
upload: '/var/vroom/sentry-profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' -> 's3://profiles/1/99/fffe9037e581422a9eefc37405254f53.attrs' (211 bytes in 0.0 seconds, 73.90 KB/s) [6097786 of 6097786]
Done. Uploaded 94406925034 bytes in 29262.6 seconds, 3.08 MB/s.
Migration completed.
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?lifecycle (timed out)
WARNING: Waiting 15 sec...
ERROR: S3 Temporary Error: Request failed for: /?lifecycle. Please try again later.
Error in install/bootstrap-s3-profiles.sh:112.
'$s3cmd --access_key=sentry --secret_key=sentry --no-ssl --region=us-east-1 --host=localhost:8333 --host-bucket='localhost:8333/%(bucket)' setlifecycle /tmp/profiles-lifecycle-policy.xml s3://profiles' exited with status 75
-> ./install.sh:main:43
--> install/bootstrap-s3-profiles.sh:source:112
```

Clearly, trying to migrate ~ 6.1 million files into s3 (took roughly 12 hours) and then expecting to apply lifecycle policy within a few minutes was quite naive.

Install script should probably stop executing and offer a proper solution when so many files found in vroom volume. Or at least continue the upgrade after s3cmd failed to apply lifecycle policy, or even do it in the background without waiting for the result.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.