lablup / lablup/callosum

Refactor streaming transports and channels

Open
#7 5 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
Python
Stars
21
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Now channels and transports are coupled together.

Currently we have the following modules:
* `callosum.lower.dispatch_redis`
- `RedisStreamAddress`
- `DispatchRedis{Connection,Connector,Binder,Transport}`
* `callosum.lower.rpc_redis`
- `RedisStreamAddress`
- `RPCRedis{Connection,Connector,Binder,Transport}`
* `callosum.pubsub.{message,channel}`
- `StreamMesasge`
- `Publisher`
- `Consumer`

One of the problem is that Redis' STREAM APIs are too complicated for our use cases.
We just need two communication patterns as in Backend.AI v19.09's event bus: fan-out broadcast & shared pipeline queue (without grouping and acks, and topic-based subscription routing can be simply done in Python instead of Redis).
Let's make the detailed grouped message streaming as a future work.
Also, the current proposal does not take "ack" into account, but later we could do it with the Redis STREAM API.

This should be refactored into:
* `callosum.lower.redis`
- `Redis{Pipeline,PubSub}Address`
- `RedisBaseTransport`
- `RedisPipeline{Connection,Connector,Binder,Transport}`
- `RedisPubSub{Connection,Connector,Binder,Transport}`
- `RedisRPC{Connection,Connector,Binder,Transport}`
* `callosum.stream.{message,channel}`
- `StreamMesage` which contains the topic, the timestamp, and the body.
- `Publisher`, `Subscriber` (fan-out broadcasting)
- `Producer`, `Consumer` (shared queue aka pipeline)

And the `stream` module should be compatible with:
* `callosum.lower.zeromq`
- ZeroMQ's pub/sub connection, connector, binder, and transport
- ZeroMQ's push/pull connection, connector, binder, and transport

We also need to let "lower" modules to provide a factory function that returns recommended combination of transports for different types of channels.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.