Federation (matrix): incoming media message (m.video/image/…) is silently dropped when remote media returns a transient 404 — no retry, event is permanently consumed

Description

On the receiving side of a Matrix-federated room, an incoming m.room.message event of type m.video / m.image / m.audio / m.file is silently dropped when the media download from the origin homeserver fails with a transient 404 Not Found. There is no retry, no backoff, and no placeholder saved — the Matrix event is marked as "unstaged" (permanently consumed) the moment the message listener swallows the error, so the attachment never appears in the receiving room and is never re-attempted, even after a pod restart.

We observed this in production on a video file (m.video) sent over federation. Image (m.image) attachments in the same federated room appear to be received correctly in normal use — which suggests a timing window that smaller images routinely clear but larger media (videos) can hit. See the Hypothesis section.

Steps to reproduce

Two Rocket.Chat servers A (sender) and B (receiver), federated via the Matrix-based federation (Enterprise), both running 8.6.0.

1. From a user on server A, send a media message (we observed it with m.video; any media msgtype should hit the same path) into a federated room.
2. On server B, the message event is received and StagingAreaService logs Processing event eventId=….
3. Server B immediately calls MatrixMediaService.downloadAndStoreRemoteFile → federationSDK.downloadFromRemoteServer, which tries three endpoints in sequence:
   • /_matrix/federation/v1/media/download/{mediaId} (failure logged at debug only)
   • /_matrix/media/v3/download/{serverName}/{mediaId}
   • /_matrix/media/r0/download/{serverName}/{mediaId} (legacy fallback)
4. If the origin returns 404 for all attempted endpoints (e.g. the media is not yet retrievable from the origin — see Hypothesis), the message listener catches the error, logs Error processing Matrix message, and returns normally.
5. StagingAreaService.processEventForRoom then calls markEventAsUnstaged(event) — the event is permanently removed from the staging queue.
6. No message is persisted in the receiving room. No retry is ever attempted, including across pod restarts.

For a deterministic reproduction of step 4, the easiest is to make the origin server return 404 for the media endpoint for a short window after the m.room.message event is sent (e.g. by delaying the upload-store commit, or by sending the message event referencing an mxc:// URI before the corresponding media upload has been committed on the origin).

Expected behavior

• A 404 (M_NOT_FOUND) from the origin's media endpoint should be treated as a transient failure for media referenced by a freshly-arrived m.room.message event, and the receiver should retry with exponential backoff (the SDK already ships a retry/backoff facility used for outgoing transactions — FEDERATION_OUTGOING_MAX_RETRIES, FEDERATION_OUTGOING_INITIAL_BACKOFF_MS, FEDERATION_OUTGOING_MAX_BACKOFF_MS, FEDERATION_OUTGOING_BACKOFF_MULTIPLIER).
• The Matrix event should not be marked as "unstaged" while the media download is still failing — it should remain staged and be retried, with the existing per-event "tried N times, removing from staging area" guard (G9/got counter in the SDK) acting as the upper bound.
• At minimum, persist a placeholder message so the receiving user is aware that something was sent and could be re-fetched later.

Actual behavior

• The homeserver.matrix.message listener catches the media-download error itself (logs Error processing Matrix message) and returns normally.
• Because the listener does not rethrow, notify() returns normally, and processEventForRoom calls markEventAsUnstaged(event) — the event is consumed without ever persisting a Rocket.Chat message.
• The SDK's retry logic only triggers for TimeoutError (J8, thrown when a queue item takes more than FEDERATION_QUEUE_MAX_TIME_PER_ROOM = 30s to process). All other errors are logged at error level and the queue item is not re-added.
• The SDK has no Matrix /sync-based backfill on startup (Rocket.Chat federation is pure PDU/EDU push), so a pod restart does not replay the failed event.

Server Setup Information

• Version of Rocket.Chat Server: 8.6.0 (helm chart rocketchat-7.0.0, image registry.rocket.chat/rocketchat/rocket.chat:8.6.0)
• License Type: Enterprise (Matrix-based federation)
• Number of Users: N/A
• Operating System: Linux (Kubernetes)
• Deployment Method: Helm
• Number of Running Instances: 2 Rocket.Chat pods (+ ddp-streamer, presence, account, authorization, nats)
• DB Replicaset Oplog: yes
• NodeJS Version: v22.22.1
• MongoDB Version: (managed by mongodb-kubernetes-operator)
• Bundled package versions on the running image:
  • @rocket.chat/federation-matrix 0.1.6
  • @rocket.chat/federation-sdk 0.6.3

Client Setup Information

N/A — server-side federation path.

Additional context

Timeline of one occurrence (timestamps UTC, identifiers redacted)

T+0.000s  StagingAreaService   "Processing event" eventId={redacted}
T+3.130s  FederationRequestService "Federation request failed"
                              url=…/_matrix/media/v3/download/{serverName}/{mediaId}
                              status=404  errorText="404 Not Found"
                              (response from a Cloudflare-fronted origin: cf-cache-status=DYNAMIC,
                               content-type=text/plain; charset=UTF-8)
T+3.130s  FederationRequestService "Federation request failed"
                              url=…/_matrix/media/r0/download/{serverName}/{mediaId}
                              status=404  (legacy fallback, also 404)
T+3.130s  federation-matrix:media-service  "Error downloading and storing remote file"
                              err.message = "Failed to download media {mediaId} from {serverName}"
                              thrown from MatrixMediaService.downloadAndStoreRemoteFile
T+3.131s  federation-matrix:message        "Error processing Matrix message"
                              same underlying error; catch block in the
                              'homeserver.matrix.message' listener swallows it

The ~3-second gap between the message event being processed and the download attempt failing is consistent with an immediate, inline fetch — the receiving server does not wait or back off before trying to fetch the referenced media.

Code path (verified against develop)

The behavior is present in both 8.6.0 and the current develop branch. On develop:

1. ee/packages/federation-matrix/src/events/message.ts — the homeserver.matrix.message listener wraps FederationMatrix.saveFederationMessage(event) in try { … } catch (err) { logger.error({ msg: 'Error processing Matrix message', err }); }. No rethrow.
2. ee/packages/federation-matrix/src/FederationMatrix.ts (saveFederationMessage, around line 1160) — calls handleMediaMessage(…) before Message.saveMessageFromFederation(…). If handleMediaMessage throws (because the download failed), no Rocket.Chat message is ever saved.
3. ee/packages/federation-matrix/src/services/MatrixMediaService.ts (downloadAndStoreRemoteFile) — calls federationSDK.downloadFromRemoteServer(serverName, mediaId) and rethrows on failure. No retry inside the service.
4. @rocket.chat/federation-sdk@0.6.3 (only ships minified; the relevant calls were observed as):
   • downloadFromRemoteServer iterates three endpoints in for await and only logs per-endpoint failures at debug level — the only thing that surfaces at error level is the final aggregate throw "Failed to download media {mediaId} from {serverName}".
   • StagingAreaQueue.processQueueItem (in federation-sdk) only re-queues items that throw TimeoutError (class J8, thrown when an event takes more than FEDERATION_QUEUE_MAX_TIME_PER_ROOM seconds to process). All other errors propagate up to processQueue's outer catch which logs Error processing item and does not re-add the item to the queue.
   • StagingAreaService.processEventForRoom (in federation-sdk) calls await this.eventService.notify({ eventId, event }) and then await this.eventService.markEventAsUnstaged(event) inside the same try block. Because the listener swallows the error, notify() resolves normally, and markEventAsUnstaged runs — the event is permanently consumed.
   • The SDK has retry/backoff machinery (retryCount, nextRetryAt, retryConfig) but it is only wired into the outgoing PerDestinationQueue, not into inbound media downloads.

Hypothesis — why images work but videos don't

Per the Matrix client-server spec, a client uploads the binary to the media endpoint, receives an mxc:// URI, and then sends the m.room.message event with url = mxc://…. For large files (videos in particular) there is a non-trivial window between "the origin's media endpoint returns an mxc:// URI" and "the bytes are committed and retrievable from the origin's storage / CDN". A federation partner that receives the m.room.message event during that window will hit a 404 on the media download.

This is consistent with what we observe:

• The receiving server attempts the download within ~3 seconds of receiving the message event.
• Images, which are typically small and commit synchronously, do not hit the window in practice.
• Videos, which take longer to commit (and may have CDN propagation delays on the origin side), do.

Even if the root cause on the origin side turns out to be something else (deleted media, expired uploads, CDN misconfiguration, etc.), the receiver's behavior is still wrong: a transient 404 on an inbound media URL should not silently lose the entire m.room.message event.

Relevant logs

See timeline above. Full log lines (with identifiers redacted) are available on request.

Suggested fix (any subset would be an improvement)

1. Stop swallowing errors in the homeserver.matrix.message listener (or at least rethrow transient ones), so the existing per-event retry counter (got / G9 — "Event … has been tried N times, removing from staging area") has a chance to work.
2. In MatrixMediaService.downloadAndStoreRemoteFile, retry downloadFromRemoteServer on 404 / 408 / 429 / 5xx with exponential backoff before giving up. A 404 on a freshly-arrived media message is far more likely to be a transient "not yet committed" state than a permanent "media does not exist".
3. Persist a placeholder Rocket.Chat message ("media not yet available") and update it once the download succeeds, so the receiving user at least sees that something was sent.
4. Surface all three endpoint attempts in downloadFromRemoteServer at the same log level (currently the first attempt's failure is debug only, making it look like only two endpoints were tried when reading the error logs).

Start with ee/packages/federation-matrix/src/events/message.ts and trace saveFederationMessage in ee/packages/federation-matrix/src/FederationMatrix.ts into MatrixMediaService.downloadAndStoreRemoteFile. Then inspect the federation-sdk staging queue behavior described in the issue. Done means a transient inbound media failure does not silently consume the Matrix event, and retry or placeholder behavior is covered by the existing staging limits.