ecosyste-ms / ecosyste-ms/roadmap
Federation, decentralization, and sync support for Ecosyste.ms data
- Dominant language
- No language data
- Stars
- 22
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
I’d like to open a discussion around supporting federation, decentralization, and syncable mirrors of Ecosyste.ms data.
## Motivations
Some possible use cases:
- Offline/private mirrors – for research, resilience, or local performance.
- Running against internal or alternative sources – e.g., scanning a private Git host or custom package registry and publishing that metadata for others to sync.
- Collaborative enrichment – enabling external contributors to generate and sync back enriched metadata, instead of relying solely on a central source.
## Federation options
There are existing protocols worth considering, such as:
- ActivityPub – general-purpose and supported by a wide range of tools.
- [ForgeFed](https://forgefed.org/) – an ActivityPub extension aimed specifically at federating forge data (issues, repos, etc).
- rsync or Git-style snapshot syncing – simpler to implement, but not so easy to handle over the web with lots of little updates.
## Questions and challenges
- What data should be federated — full database dumps, deltas, or specific datasets (e.g., per ecosystem)?
- How to deal with large-scale data (e.g. all npm metadata or dependent repo feeds)?
- What trust and verification mechanisms are needed for federated mirrors?
- How would we structure the sync protocol for different use cases (one-way mirroring, two-way sync, etc)?
- Can we make federation opt-in and ecosystem-specific?
- How to support sync across non-public or internal instances?
- Some of the ecosyste.ms services have TBs of data in them, syncing them whole is unfeasible at the moment from a performance point of view, but also unlikely many users are going to want the whole thing, only the bits relevant to them.
This is early thinking — I’m opening this up to hear ideas, use cases, potential protocols etc.
Contributor guide
Assessment
This issue has not been assessed yet.