clearlydefined / clearlydefined/service

Need way to mirror definitions

Open
#386 33 comments 3 reactions 0 assignees View on GitHub
design enhancement GSoC-Candidate
Dominant language
TypeScript
Stars
51
Forks
46
PR merge metrics
No merged PRs in 30d

Description

There is interest in people replicating definition data locally to support robustness/performance, infrastructure control, and privacy.

Principles:
* Readonly -- clearlydefined.io is still the source of truth with all curations, harvesting, ... done in the production service.
* Definitions only -- Harvested data and curations are not included in the mirroring process.
* Only point queries -- The production service supports arbitrary queries over definition documents. The local copy only needs point queries based on component coordinates.

Options:
* rsync-style -- The definitions are just blobs so in theory we could mirror those as files and allow people to read from disk. That exposes the user to internal details of ClearlyDefined.
* slave service -- Implement a path through the service code that is read-only and has the mirroring activity built in. This would shut down any write paths, not have a crawler, ... and implement whatever mirroring protocol we decide is best.

Random thoughts/topics
- [ ] Must all definitions be aggressively computed? Currently we (re)compute definitions on demand in the event of schema changes. We could have the local service fall back to the remote service if the schemas don't match.
- [ ] First replication is different. That could be a bulk download of a dump where as keeping up to date continuously replicates recent actions.
- [ ] periodic or continuous. Need to determine if the use cases require up to the minute replication or if periodic (hourly, daily) replication is enough.
- [ ] should be related to the need for an "event stream" that enables people to track new definitions.
- [ ] Local scenarios may use different data store technology from the main service. A simple version would just put the data in the local file system. So this is not a straight record for record mirror. Rather it should use an API to read and write the data using the correct structures.
- [ ] Local servers, being read-only, need not ever compute a definition.

cc: @jeffmendoza

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by turning the listed options and open questions into a decided design: define the initial and ongoing replication protocol, readonly definitions-only behavior, point-query interface, and storage assumptions. Done means the mirroring approach and its required behavior are specified clearly enough for implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.