Cleanup old data

No files, tests, or entry points are named. Review the existing Sentry cleanup job and retention behavior for sentry_auditlogentry, sentry_file, sentry_fileblob, and sentry_userip, then determine which tables need configurable retention and how the duplicate-index upgrade cases relate. Done requires an agreed table scope, retention policy, and upgrade-safe behavior.

As part of the self hosted upgrade to 26.9.0 I have encountered quite a few of unique constraint errors on re-index.

For now I was able to fix those errors manually, but in doing so I noticed, that there were a few tables in the postgres instance, that had data present which was 2+ years old - which I don't think is necessary to keep.

Here are some of my errors I encountered while doing the upgrade:

ERROR:  could not create unique index "sentry_release_organization_id_version_65da5843_uniq"
DETAIL:  Key (organization_id, version)=(1, Zi8Qa7e9j_dtyKR4GOrq3) is duplicated.

postgres=# SELECT id, version, date_added, user_agent FROM sentry_release WHERE version='Zi8Qa7e9j_dtyKR4GOrq3';
  id  |        version        |          date_added           |               user_agent                
------+-----------------------+-------------------------------+-----------------------------------------
 4819 | Zi8Qa7e9j_dtyKR4GOrq3 | 2024-08-30 07:44:11.197817+00 | sentry-cli/1.77.3 webpack-plugin/1.21.0
 4818 | Zi8Qa7e9j_dtyKR4GOrq3 | 2024-08-30 07:44:10.840451+00 | sentry-cli/1.77.3 webpack-plugin/1.21.0

and

ERROR:  could not create unique index "unique_org_string"
DETAIL:  Key (string, organization_id)=(d4d4d88fc37dcd5e17ea30a4074b9c98065280b4, 1) is duplicated.

postgres=# SELECT *
FROM sentry_stringindexer
WHERE string IN (
    SELECT string
    FROM sentry_stringindexer
    GROUP BY string
    HAVING COUNT(*) > 1
)
ORDER BY string;
 id  |                  string                  | organization_id |          date_added           |           last_seen           | retention_days 
-----+------------------------------------------+-----------------+-------------------------------+-------------------------------+----------------
 400 | d4d4d88fc37dcd5e17ea30a4074b9c98065280b4 |               1 | 2025-06-18 12:06:59.306133+00 | 2025-06-18 12:06:59.30614+00  |             90
 401 | d4d4d88fc37dcd5e17ea30a4074b9c98065280b4 |               1 | 2025-06-18 12:07:10.29119+00  | 2025-06-18 12:07:10.291192+00 |             90
(2 rows)

after doing this cleanup I also checked all tables present in postgres and can now provide this list, which also contains +2 year old data (no idea if this is needed for something)

• sentry_auditlogentry (5.000 rows)
• sentry_file (10.000 rows)
• sentry_fileblob (10.000 rows)
• sentry_userip (2.000 rows)

All of these have data still present from 3+ years ago

There were a few other talbes with old data as well, but they just had a few hundred rows if even.

Maybe this should be part of the sentry cleanup job to remove data from certain tables after they a configurable amount of time has passed.

I will re-check this in like 6 months after my instance re-fills with events and data and - if needed - will create a new issue then with the new info related to the other tables.