AllenNeuralDynamics / AllenNeuralDynamics/one-liner

SharedMemory Implementation for "Big" Data

Offen
#32 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
3
Forks
0
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

Because zmq does not have a shared memory transport layer across processes, it could be useful to implement our own and wrap it under the same one-liner API. This would be especially useful for passing around camera data locally.

Because the data size can cause issues if we let it grow unbounded, we would need to be pickier about how we deal with subscribers. Specifically, we would need to know how to buffer data on the sender side depending on how subscribers are asking for the data.

_Simple Case_
```python
# Setup 2 subscribers who both want the latest camera frame when they ask for it.
client0.configure_stream("camera_frame", storage_type="cache")
client1.configure_stream("camera_frame", storage_type="cache")
```
_Publisher Buffering Case_
```python
# Setup a subscriber to get all camera frames
client2.configure_stream("camera_frame", storage_type="queue")
```

_Publisher Buffering and tracking Subscriber Case_
This would be especially challenging if multiple subscribers are asking the server to buffer data for them while reading them at different rates.
```python
# With 2 subscribers, now the publisher needs to maintain an index of which subscriber got what.
client3.configure_stream("camera_frame", storage_type="queue").
```

The final layer of complexity is that subscribers need to tell the publisher when they're "done" with the data **if** they aren't immediately copyiing the data out of shared memory.

```python
frame = client0.get_stream("camera_frame", copy=True) # OK! we can release shared memory.
# and tell the publisher we're done with it.

frame = client0.get_stream("camera_frame", copy=False) # Hmm. Do we use a Context Manager here?'
# How do we release shared memory and
# tell the "publisher" we're done with it?
```

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.