AllenNeuralDynamics / AllenNeuralDynamics/one-liner

SharedMemory Implementation for "Big" Data

オープン
#32 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
3
フォーク
0
PR マージ指標
30日以内にマージされた PR はありません

説明

Because zmq does not have a shared memory transport layer across processes, it could be useful to implement our own and wrap it under the same one-liner API. This would be especially useful for passing around camera data locally.

Because the data size can cause issues if we let it grow unbounded, we would need to be pickier about how we deal with subscribers. Specifically, we would need to know how to buffer data on the sender side depending on how subscribers are asking for the data.

_Simple Case_
```python
# Setup 2 subscribers who both want the latest camera frame when they ask for it.
client0.configure_stream("camera_frame", storage_type="cache")
client1.configure_stream("camera_frame", storage_type="cache")
```
_Publisher Buffering Case_
```python
# Setup a subscriber to get all camera frames
client2.configure_stream("camera_frame", storage_type="queue")
```

_Publisher Buffering and tracking Subscriber Case_
This would be especially challenging if multiple subscribers are asking the server to buffer data for them while reading them at different rates.
```python
# With 2 subscribers, now the publisher needs to maintain an index of which subscriber got what.
client3.configure_stream("camera_frame", storage_type="queue").
```

The final layer of complexity is that subscribers need to tell the publisher when they're "done" with the data **if** they aren't immediately copyiing the data out of shared memory.

```python
frame = client0.get_stream("camera_frame", copy=True) # OK! we can release shared memory.
# and tell the publisher we're done with it.

frame = client0.get_stream("camera_frame", copy=False) # Hmm. Do we use a Context Manager here?'
# How do we release shared memory and
# tell the "publisher" we're done with it?
```

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。