matrix-org / matrix-org/matrix-viewer

Fetch events from multiple homeservers

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
A-archive-room-view A-backend T-Enhancement
Dominant language
JavaScript
Stars
84
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Problem

When using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint, it creates events all at the same `depth` and are only correctly sorted between because of the `stream_ordering`. This works great for the local homeserver as they are all in the correct order but can get sticky when federating to other homeservers because `stream_ordering` is just a local number and not federated like `depth`. If other homeservers always backfilled in order from newest -> oldest like scrollbar in a room, everything would be fine. But...

> [...] it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
> - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
> - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
>
> *-- https://github.com/matrix-org/synapse/pull/11114*

This is especially important for the `gitter.im` homeserver because we will use MSC2716 to backfill the entire message history from Gitter to Matrix as part of the [sunset plan](https://github.com/vector-im/roadmap/issues/26). We want the messages to be in the correct order and available correctly in the archive. Pulling directly from the `gitter.im` homeserver will allow us to bide our time until we get to solving graph linearization the correct way via online topological ordering.

## Potential solutions

Instead of pulling from a single `config.matrixServerUrl` with `config.matrixAccessToken`, we can have a map from homeserver to access token and if a given room alias matches a homeserver we have the access token for, we can pull from it directly. This saves time from not having to join over federation in those cases but also very important for the MSC2716 problem described above.

```json
{
"matrixServerSecretMap": {
"matrix.org": "xxx",
"gitter.im": "xxx"
}
}
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the configuration and event-fetching entry points that use config.matrixServerUrl and config.matrixAccessToken. Review how room aliases are matched and how MSC2716 history is fetched, then compare the behavior with the proposed matrixServerSecretMap. Done means configured homeservers can be selected for matching room aliases and events are fetched in the correct order.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.