Replay recording blobs are not deleted after retention expires on filesystem storage

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

1. Deploy Sentry self-hosted version 26.6.0.

2. Use the filesystem filestore configuration without configuring a separate Replay storage backend:

   filestore.backend: filesystem
   filestore.options:
     location: /data/files
   

   replay.storage.backend is not configured, so Replay recordings are stored in the main filestore.

3. Configure the global Sentry retention period:

   SENTRY_EVENT_RETENTION_DAYS=90
   

4. Enable Session Replay for one or more projects and continuously ingest Replay data for longer than 90 days.

5. Verify that the cleanup container runs daily with the configured retention:

   docker inspect sentry-self-hosted-sentry-cleanup-1 \
     --format '{{range .Config.Env}}{{println .}}{{end}}' |
   grep SENTRY_EVENT_RETENTION_DAYS
   
   docker inspect sentry-self-hosted-sentry-cleanup-1 \
     --format 'CMD={{json .Config.Cmd}} ENTRYPOINT={{json .Config.Entrypoint}}'
   

   Output:

   SENTRY_EVENT_RETENTION_DAYS=90
   CMD=["0 0 * * * gosu sentry sentry cleanup --days 90"]
   

6. Verify that Replay metadata uses the expected retention in ClickHouse:

   docker exec sentry-self-hosted-clickhouse-1 \
     clickhouse-client --query "
   SELECT
       retention_days,
       count() AS rows,
       uniqExact(replay_id) AS replays,
       min(timestamp) AS oldest,
       max(timestamp) AS newest
   FROM default.replays_local
   GROUP BY retention_days
   ORDER BY retention_days
   "
   

   The Replay rows have:

   retention_days = 90
   

7. Verify the TTL configured on the Replay table:

   docker exec sentry-self-hosted-clickhouse-1 \
     clickhouse-client --query "
   SHOW CREATE TABLE default.replays_local
   "
   

   The table definition contains:

   TTL timestamp + toIntervalDay(retention_days)
   

8. Verify that Replay metadata older than its retention period is being removed from ClickHouse:

   docker exec sentry-self-hosted-clickhouse-1 \
     clickhouse-client --query "
   SELECT
       count() AS expired_rows,
       uniqExact(replay_id) AS expired_replays
   FROM default.replays_local
   WHERE timestamp + toIntervalDay(retention_days) < now()
   "
   

   Only a small number of rows waiting for a ClickHouse TTL merge remain expired.

9. Check the filesystem used by the main Sentry filestore:

   V=/var/lib/docker/volumes/sentry-data/_data/files
   
   du -sh "$V"
   find "$V" -xdev -type f | wc -l
   

   In the affected installation:

   approximately 234 GiB
   4,492,527 files
   

10. Calculate the number and size of files older than the configured 90-day retention:

    find "$V" -xdev -type f -mtime +90 -printf '%s\n' |
    awk '
    {
        bytes += $1
        files++
    }
    END {
        printf "files older than 90d: %d\n", files
        printf "size older than 90d: %.2f GiB\n", bytes/1024/1024/1024
    }'
    

    In the affected installation:

    files older than 90d: 3,961,730
    size older than 90d: 202.65 GiB
    

11. Inspect several remaining files:

    find "$V/90" -type f | head
    

    The files use the Replay direct-storage path format:

    /data/files/<retention_days>/<project_id>/<replay_id>/<segment_id>
    

    Example:

    /data/files/90/12/cc2a622a8b48485d94f3ae9b589f5232/0
    /data/files/90/12/cc2a622a8b48485d94f3ae9b589f5232/1
    

12. Verify that these files are not represented by regular Sentry FileBlob records:

    docker exec -i sentry-self-hosted-postgres-1 \
      psql -U sentry -d sentry -P pager=off -c "
    SELECT
        COUNT(*) AS blobs,
        pg_size_pretty(COALESCE(SUM(size), 0)::bigint) AS total_size
    FROM sentry_fileblob;
    "
    

    In the affected installation, PostgreSQL contains only 13 regular blobs totaling approximately 5.4 MiB.

13. Verify that the legacy Replay segment table is empty:

    docker exec -i sentry-self-hosted-postgres-1 \
      psql -U sentry -d sentry -P pager=off -c "
    SELECT COUNT(*)
    FROM replays_replayrecordingsegment;
    "
    

    Output:

    0
    

14. Review the regular cleanup logs:

    docker logs --since 168h \
      sentry-self-hosted-sentry-cleanup-1 2>&1 |
    grep -E 'ReplayRecordingSegment|Clean up took'
    

    The cleanup completes but repeatedly reports:

    Removing ReplayRecordingSegment for days=90 project=*
    [SCHEDULED] No ReplayRecordingSegment objects found to delete
    

15. Run sentry cleanup --days 90 again or wait for another scheduled cleanup execution.

16. Check /data/files again. Replay metadata continues to expire from ClickHouse, but physical Replay recording segments older than 90 days remain on the filesystem.

Expected Result

When a Session Replay reaches its configured retention period, Sentry should automatically remove both the Replay metadata and all associated physical recording segments.

Specifically:

1. Replay metadata should expire from Snuba/ClickHouse according to the Replay's retention_days value.

2. All associated RRWeb recording segments should be deleted from the configured Replay storage backend.

3. When the default filesystem backend is used, files stored under paths such as:

   /data/files/<retention_days>/<project_id>/<replay_id>/<segment_id>
   

   should be removed automatically after the retention period expires.

4. The self-hosted cleanup or another scheduled Sentry maintenance task should support Replay recordings stored using the current direct-storage format.

5. Expiring Replay metadata must not leave the corresponding recording blobs orphaned in the filesystem.

6. Filesystem usage should decrease after expired Replay recordings are removed and should not grow indefinitely because of Replay data older than the configured retention.

7. Sentry should provide a supported maintenance or recovery command that can detect and remove orphaned Replay recording blobs when the corresponding Snuba/ClickHouse metadata has already expired.

8. Administrators should not need to delete files directly from the Docker volume.

Actual Result

Replay metadata expires correctly from ClickHouse, but the associated physical Replay recording segments remain in the filesystem.

The affected installation currently contains:

/data/files size:             approximately 234 GiB
total file count:             4,492,527
files older than 90 days:     3,961,730
size older than 90 days:      approximately 202.65 GiB

The host filesystem has reached:

Filesystem      Size  Used  Avail  Use%
/dev/sda1       301G  274G    15G   95%

Most of the storage is used by paths following the Replay direct-storage format:

/data/files/90/<project_id>/<replay_id>/<segment_id>

Example:

/data/files/90/12/cc2a622a8b48485d94f3ae9b589f5232/0
/data/files/90/12/cc2a622a8b48485d94f3ae9b589f5232/1

Some of these Replay recording segments have filesystem modification timestamps several months older than the configured 90-day retention.

ClickHouse is configured correctly:

TTL timestamp + toIntervalDay(retention_days)

Only a very small number of expired Replay rows remain while waiting for normal ClickHouse TTL merges. This indicates that Replay metadata retention is working.

The regular Sentry file tables do not account for the filesystem usage:

sentry_fileblob:
13 objects
approximately 5.4 MiB

The legacy Replay recording table is also empty:

replays_replayrecordingsegment:
0 rows

The daily cleanup finishes successfully but reports:

Removing ReplayRecordingSegment for days=90 project=*
[SCHEDULED] No ReplayRecordingSegment objects found to delete

The cleanup therefore checks the empty legacy Replay model but does not delete the current direct-storage Replay blobs.

Sentry includes internal Replay deletion tasks that remove recording segments through the configured storage backend. However, the available deletion process first queries Replay metadata from Snuba/ClickHouse. Once the metadata has already expired through the ClickHouse TTL, the old filesystem blobs can no longer be discovered by that process.

As a result, Replay metadata is deleted after 90 days, while the corresponding physical recording segments remain indefinitely and continue consuming disk space.

Product Area

Replays

Link

No response

DSN

No response

Version

26.6.0