Synapse sometimes fails to deliver media

### Description

Hello,

Our Matrix instance sometimes fails to deliver media to our Element web client (v1.12.0, though the bug happened prior to this version).

Media loads well most of the time. But **several times a day**, in a seemingly random fashion, Synapse would respond 404 to all existing medias, both remote and local. The issue seems to solve itself after a few minutes.

We did not test with other clients. Authenticated media feature is enabled (I suppose it may be related).

I don’t know if the culprit is Synapse or Element, but I would say Synapse since the query seems to be valid on the Element side.

### Steps to reproduce

The bug is very inconsistent, thus quite hard to reproduce.

- Open an image from the Element web client.
- The client will likely issue this kind of request: `GET https://matrix.lacontrevoie.fr/_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true`
- If the bug triggers, the image just won’t display in Element. In the dev console, Synapse responds 404 − `{"errcode":"M_NOT_FOUND","error":"Not found"}` and will do so for all media during a few minutes.
- If you refresh the page during this time (or even clear the Element cache and reload), all users and room avatars will disappear, as well as the image thumbnails. Synapse will respond 404 for all media.
- Wait a few minutes.
- Try to open the same image again. The very same request and media ID will be queried by the client, though Synapse will respond 200 with the image.

Here’s a filtered view of Firefox’s development console, network tab:

### Homeserver

matrix.lacontrevoie.fr

### Synapse Version

1.138.2

### Installation Method

Docker (matrixdotorg/synapse)

### Database

Fresh install from PostgreSQL 17.6

### Workers

Single process

### Platform

Host: Debian Trixie, Linux 6.12.41, AMD64 architecture, dedicated server with 64GB RAM, Docker 28.4.0.

### Configuration

We are using MAS and a few other stuff (EC RTC setup, appservices, sticker widget…).
The used room is **not encrypted** and uses the default media policy.
Presence is disabled.

Media retention uses the following policy:
```
retention:
enabled: true
default_policy:
max_lifetime: 2y
allowed_lifetime_min: 7d
allowed_lifetime_max: 3y

media_retention:
local_media_lifetime: 1y
remote_media_lifetime: 180d
```

The following experimental features are enabled:
```
experimental_features:
msc3266_enabled: true
msc4222_enabled: true
msc4140_enabled: true
msc3861:
enabled: true
```

The following module is enabled: [synapse_http_antispam.HTTPAntispam](https://github.com/maunium/synapse-http-antispam)

The following appservices are used and enabled:
- [mautrix-discord](https://github.com/mautrix/discord/)
- [draupnir](https://github.com/the-draupnir-project/Draupnir/)

We are **not** using any storage provider such as [S3](https://github.com/matrix-org/synapse-s3-storage-provider). The media is directly stored on block storage, the classic way.

### Relevant log output

```shell
Grep on the Synapse logs with the media ID. Censored some data.
There is a huge time gap between the 404 and the 200 response (30 minutes) because I was in the middle of writing this ticket, though the issue usually solves itself in about 5 minutes.

2025-09-28 15:22:58,549 - synapse.http.server - 133 - INFO - GET-118417 - method='GET' uri='/_matrix/media/v3/thumbnail/lacontrevoie.fr/?width=800&height=600&method=scale&allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:22:58,550 - synapse.access.http.8008 - 515 - INFO - GET-118417 - - 8008 - {None} Processed request: 0.003sec/0.001sec (0.001sec, 0.001sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/thumbnail/lacontrevoie.fr/?width=800&height=600&method=scale&allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:22:58,649 - synapse.http.server - 133 - INFO - GET-118453 - method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:22:58,649 - synapse.access.http.8008 - 515 - INFO - GET-118453 - - 8008 - {None} Processed request: 0.001sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:23:06,442 - synapse.http.server - 133 - INFO - GET-118458 - method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:23:06,444 - synapse.access.http.8008 - 515 - INFO - GET-118458 - - 8008 - {None} Processed request: 0.003sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:23:12,419 - synapse.http.server - 133 - INFO - GET-118460 - method='GET' uri='/_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true' clientproto='HTTP/1.1' site='8008'> SynapseError: 404 - Not found
2025-09-28 15:23:12,420 - synapse.access.http.8008 - 515 - INFO - GET-118460 - - 8008 - {None} Processed request: 0.002sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.001sec/1) 45B 404 "GET /_matrix/media/v3/download/lacontrevoie.fr/?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
2025-09-28 15:57:23,103 - synapse.access.http.8008 - 515 - INFO - GET-119245 - - 8008 - {@neil:lacontrevoie.fr} Processed request: 0.007sec/-0.000sec (0.001sec, 0.001sec) (0.000sec/0.001sec/1) 256088B 200 "GET /_matrix/client/v1/media/download/lacontrevoie.fr/?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64; rv:143.0) Gecko/20100101 Firefox/143.0" [0 dbevts]
```

### Anything else that would be useful to know?

- Default and tested room version : 12
- Authenticated media feature enabled

Reverse-proxy (Caddy) configuration is the following (irrelevant lines omitted):
```
matrix.lacontrevoie.fr {
header Content-Security-Policy "default-src 'self' 'unsafe-inline';"

route {
# unrelated
header /.well-known/* {
...
}

# forward authentication to MAS
@matrix_auth path_regexp ^/_matrix/client/(.*)/(login|logout|refresh)
reverse_proxy @matrix_auth

# forward reports to Draupnir
@matrix_reports path_regexp re_reports ^/_matrix/client/(r0|v3)/rooms/(?[^/]*)/report/(?.*)$
handle @matrix_reports {
rewrite /api/1/report/{re.re_reports.room_id}/{re.re_reports.event_id}
reverse_proxy {
to
header_down ...
}
}

# Forward the rest to Synapse
reverse_proxy
}
}
```