AllenNeuralDynamics / AllenNeuralDynamics/one-liner
SharedMemory Implementation for "Big" Data
- Lenguaje dominante
- Python
- Estrellas
- 3
- Forks
- 0
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Because zmq does not have a shared memory transport layer across processes, it could be useful to implement our own and wrap it under the same one-liner API. This would be especially useful for passing around camera data locally.
Because the data size can cause issues if we let it grow unbounded, we would need to be pickier about how we deal with subscribers. Specifically, we would need to know how to buffer data on the sender side depending on how subscribers are asking for the data.
_Simple Case_
```python
# Setup 2 subscribers who both want the latest camera frame when they ask for it.
client0.configure_stream("camera_frame", storage_type="cache")
client1.configure_stream("camera_frame", storage_type="cache")
```
_Publisher Buffering Case_
```python
# Setup a subscriber to get all camera frames
client2.configure_stream("camera_frame", storage_type="queue")
```
_Publisher Buffering and tracking Subscriber Case_
This would be especially challenging if multiple subscribers are asking the server to buffer data for them while reading them at different rates.
```python
# With 2 subscribers, now the publisher needs to maintain an index of which subscriber got what.
client3.configure_stream("camera_frame", storage_type="queue").
```
The final layer of complexity is that subscribers need to tell the publisher when they're "done" with the data **if** they aren't immediately copyiing the data out of shared memory.
```python
frame = client0.get_stream("camera_frame", copy=True) # OK! we can release shared memory.
# and tell the publisher we're done with it.
frame = client0.get_stream("camera_frame", copy=False) # Hmm. Do we use a Context Manager here?'
# How do we release shared memory and
# tell the "publisher" we're done with it?
```
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.