microsoft / microsoft/Qcodes

Data set modes

Open
#336 1 comment 2 reactions 0 assignees View on GitHub
mulitprocessing old dataset
Dominant language
Python
Stars
459
Forks
359
Avg merge
3d 6h
Merged PRs (30d)
73

Description

Currently a data set can run in one of 3 modes:
1. Local: data is stored in local memory and written to/read from disk locally
2. Push to server: data is written by sending to a data server process
3. Pull from server: data is read by fetching from a data server process
This is fine for now, but eventually will probably be too restrictive.

I think the data set mode is currently mixing a few bits of information about the data set:
1. Role: is this the authoritative, master copy of the data, or is this a shadow copy?
2. Access: does this data set allow reading? Does it allow writing?
3. Concurrency: does this data set maintain updates from other writers? Does it note and handle merge conflicts? Or does it only support a single writer?
4. Completeness: is the data set complete, or might it change during use?

Today we make some assumptions about the allowable combinations of these mode attributes. In the future, we may want to allow more combinations.

This should get folded into a broader issue covering distributed data sets, which would deal with things like "slice" data sets, multiple update sources, data sources/publishers and subscribers, data set subscribers, data set aggregation, etc., etc.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by tracing how the current local, push-to-server, and pull-from-server modes are represented, then define the role, access, concurrency, and completeness combinations needed for distributed data sets; the issue does not specify an implementation boundary or a concrete definition of done.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.