byexamples / byexamples/byexample

Share with workers a private copy of data structures set by main thread, stored on the `namespace` (`ns`) object

Open
#252 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
67
Forks
9
PR merge metrics
No merged PRs in 30d

Description

**Describe the feature you'd like**
The `namespace` or `ns` object is used to communicate or share data structures between the main thread and the workers and between the workers.
These data structures need to be created by the `sharer` in order to be, well, really shared and if they are mutable, they must also be protected with a kind of a lock (`RLock`), also created by `sharer`.

This allows a communication *back and forward* between the main thread and the workers and between the workers.

But in some cases it is needed something simpler: a main thread share information with its worker in one way fashion and the workers don't need to communicate between themselves. This scenario makes a fully N-to-N shared data structure and the requirement of a lock totally unnecessary.

In this case `byexample` could allow to create non-shared data-structures like plain `list`, `dict` or `int` in the main thread and make them visible in the workers as *private copies*.

To ensure that this will work in `threading` and in `multiprocessing` environments, the *private* objects must be pickle-able and the copy process should be `(take main's object)->pickle->(get the bytes)->unpickle->(get the copy object)`

**Additional context (optional)**
The documentation of `byexample` does not explain what happen if `ns` is used to store objects *not* created by `sharer`. In principle we could forbid them or we could make it private. In any case it may break some 3rd party plugin but the impact should be quite low.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.