ecosyste-ms / ecosyste-ms/roadmap

Federation, decentralization, and sync support for Ecosyste.ms data

Open
#28 1 comment 1 reaction 0 assignees View on GitHub
help wanted
Dominant language
No language data
Stars
22
Forks
2
PR merge metrics
No merged PRs in 30d

Description

I’d like to open a discussion around supporting federation, decentralization, and syncable mirrors of Ecosyste.ms data.

## Motivations

Some possible use cases:
- Offline/private mirrors – for research, resilience, or local performance.
- Running against internal or alternative sources – e.g., scanning a private Git host or custom package registry and publishing that metadata for others to sync.
- Collaborative enrichment – enabling external contributors to generate and sync back enriched metadata, instead of relying solely on a central source.

## Federation options

There are existing protocols worth considering, such as:
- ActivityPub – general-purpose and supported by a wide range of tools.
- [ForgeFed](https://forgefed.org/) – an ActivityPub extension aimed specifically at federating forge data (issues, repos, etc).
- rsync or Git-style snapshot syncing – simpler to implement, but not so easy to handle over the web with lots of little updates.

## Questions and challenges
- What data should be federated — full database dumps, deltas, or specific datasets (e.g., per ecosystem)?
- How to deal with large-scale data (e.g. all npm metadata or dependent repo feeds)?
- What trust and verification mechanisms are needed for federated mirrors?
- How would we structure the sync protocol for different use cases (one-way mirroring, two-way sync, etc)?
- Can we make federation opt-in and ecosystem-specific?
- How to support sync across non-public or internal instances?
- Some of the ecosyste.ms services have TBs of data in them, syncing them whole is unfeasible at the moment from a performance point of view, but also unlikely many users are going to want the whole thing, only the bits relevant to them.

This is early thinking — I’m opening this up to hear ideas, use cases, potential protocols etc.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.