Refactor streaming transports and channels
- Dominant language
- Python
- Stars
- 21
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Now channels and transports are coupled together.
Currently we have the following modules:
* `callosum.lower.dispatch_redis`
- `RedisStreamAddress`
- `DispatchRedis{Connection,Connector,Binder,Transport}`
* `callosum.lower.rpc_redis`
- `RedisStreamAddress`
- `RPCRedis{Connection,Connector,Binder,Transport}`
* `callosum.pubsub.{message,channel}`
- `StreamMesasge`
- `Publisher`
- `Consumer`
One of the problem is that Redis' STREAM APIs are too complicated for our use cases.
We just need two communication patterns as in Backend.AI v19.09's event bus: fan-out broadcast & shared pipeline queue (without grouping and acks, and topic-based subscription routing can be simply done in Python instead of Redis).
Let's make the detailed grouped message streaming as a future work.
Also, the current proposal does not take "ack" into account, but later we could do it with the Redis STREAM API.
This should be refactored into:
* `callosum.lower.redis`
- `Redis{Pipeline,PubSub}Address`
- `RedisBaseTransport`
- `RedisPipeline{Connection,Connector,Binder,Transport}`
- `RedisPubSub{Connection,Connector,Binder,Transport}`
- `RedisRPC{Connection,Connector,Binder,Transport}`
* `callosum.stream.{message,channel}`
- `StreamMesage` which contains the topic, the timestamp, and the body.
- `Publisher`, `Subscriber` (fan-out broadcasting)
- `Producer`, `Consumer` (shared queue aka pipeline)
And the `stream` module should be compatible with:
* `callosum.lower.zeromq`
- ZeroMQ's pub/sub connection, connector, binder, and transport
- ZeroMQ's push/pull connection, connector, binder, and transport
We also need to let "lower" modules to provide a factory function that returns recommended combination of transports for different types of channels.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.