AllenNeuralDynamics / AllenNeuralDynamics/one-liner
SharedMemory Implementation for "Big" Data
- Langage dominant
- Python
- Étoiles
- 3
- Forks
- 0
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
Because zmq does not have a shared memory transport layer across processes, it could be useful to implement our own and wrap it under the same one-liner API. This would be especially useful for passing around camera data locally.
Because the data size can cause issues if we let it grow unbounded, we would need to be pickier about how we deal with subscribers. Specifically, we would need to know how to buffer data on the sender side depending on how subscribers are asking for the data.
_Simple Case_
```python
# Setup 2 subscribers who both want the latest camera frame when they ask for it.
client0.configure_stream("camera_frame", storage_type="cache")
client1.configure_stream("camera_frame", storage_type="cache")
```
_Publisher Buffering Case_
```python
# Setup a subscriber to get all camera frames
client2.configure_stream("camera_frame", storage_type="queue")
```
_Publisher Buffering and tracking Subscriber Case_
This would be especially challenging if multiple subscribers are asking the server to buffer data for them while reading them at different rates.
```python
# With 2 subscribers, now the publisher needs to maintain an index of which subscriber got what.
client3.configure_stream("camera_frame", storage_type="queue").
```
The final layer of complexity is that subscribers need to tell the publisher when they're "done" with the data **if** they aren't immediately copyiing the data out of shared memory.
```python
frame = client0.get_stream("camera_frame", copy=True) # OK! we can release shared memory.
# and tell the publisher we're done with it.
frame = client0.get_stream("camera_frame", copy=False) # Hmm. Do we use a Context Manager here?'
# How do we release shared memory and
# tell the "publisher" we're done with it?
```
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Évaluation
Cette issue n'a pas encore été évaluée.