Data set modes
- Dominant language
- Python
- Stars
- 459
- Forks
- 359
- Avg merge
- 3d 6h
- Merged PRs (30d)
- 73
Description
Currently a data set can run in one of 3 modes:
1. Local: data is stored in local memory and written to/read from disk locally
2. Push to server: data is written by sending to a data server process
3. Pull from server: data is read by fetching from a data server process
This is fine for now, but eventually will probably be too restrictive.
I think the data set mode is currently mixing a few bits of information about the data set:
1. Role: is this the authoritative, master copy of the data, or is this a shadow copy?
2. Access: does this data set allow reading? Does it allow writing?
3. Concurrency: does this data set maintain updates from other writers? Does it note and handle merge conflicts? Or does it only support a single writer?
4. Completeness: is the data set complete, or might it change during use?
Today we make some assumptions about the allowable combinations of these mode attributes. In the future, we may want to allow more combinations.
This should get folded into a broader issue covering distributed data sets, which would deal with things like "slice" data sets, multiple update sources, data sources/publishers and subscribers, data set subscribers, data set aggregation, etc., etc.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by tracing how the current local, push-to-server, and pull-from-server modes are represented, then define the role, access, concurrency, and completeness combinations needed for distributed data sets; the issue does not specify an implementation boundary or a concrete definition of done.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100