AllenNeuralDynamics / AllenNeuralDynamics/one-liner

SharedMemory Implementation for "Big" Data

Đang mở
#32 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
3
Fork
0
Chỉ số merge pull request
Không có pull request nào được merge trong 30 ngày

Mô tả

Because zmq does not have a shared memory transport layer across processes, it could be useful to implement our own and wrap it under the same one-liner API. This would be especially useful for passing around camera data locally.

Because the data size can cause issues if we let it grow unbounded, we would need to be pickier about how we deal with subscribers. Specifically, we would need to know how to buffer data on the sender side depending on how subscribers are asking for the data.

_Simple Case_
```python
# Setup 2 subscribers who both want the latest camera frame when they ask for it.
client0.configure_stream("camera_frame", storage_type="cache")
client1.configure_stream("camera_frame", storage_type="cache")
```
_Publisher Buffering Case_
```python
# Setup a subscriber to get all camera frames
client2.configure_stream("camera_frame", storage_type="queue")
```

_Publisher Buffering and tracking Subscriber Case_
This would be especially challenging if multiple subscribers are asking the server to buffer data for them while reading them at different rates.
```python
# With 2 subscribers, now the publisher needs to maintain an index of which subscriber got what.
client3.configure_stream("camera_frame", storage_type="queue").
```

The final layer of complexity is that subscribers need to tell the publisher when they're "done" with the data **if** they aren't immediately copyiing the data out of shared memory.

```python
frame = client0.get_stream("camera_frame", copy=True) # OK! we can release shared memory.
# and tell the publisher we're done with it.

frame = client0.get_stream("camera_frame", copy=False) # Hmm. Do we use a Context Manager here?'
# How do we release shared memory and
# tell the "publisher" we're done with it?
```

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.