libp2p / libp2p/go-libp2p-pubsub
Topic Management Strategies, Implementation, And Questions
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 364
- Forks
- 221
- PR merge metrics
- No merged PRs in 30d
Description
# Topic Management Strategies And Implementation
One of the recent changes to this library (which I personally like, as it has resulted in much better message throughput in benchmarks) makes it so that calling `Join` on a topic more than once results in an error, which initially makes having multiple goroutines access the same topic somewhat problematic because `Join` doesn't internally store the topic, and instead means the caller needs to implement some form of topic management, allowing you to share the same topic between multiple different goroutines.
Currently I've implemented what I call a ["PubSubLocker"](https://gist.github.com/bonedaddy/1008cdfe8cc16a21d105e8db9e7bb142) that allows me to pass around the topic, or subscription handler to different goroutines. I then have a shim I wrote around `PubSub` that intercepts `Join` calls, and routes them through the `PubSubLocker`, example:
```
// Join joins the topic and returns a Topic handle.
// Only one Topic handle should exist per topic, and Join will error if
// the Topic handle already exists.
func (psx *Pubsubx) Join(topic string, opts ...ps.TopicOpt) (*ps.Topic, error) {
return psx.pslocker.GetOrSetTopic(topic, psx.pb, opts...)
}
```
It is doubtful I will be the only one to run into a situation like this, so it's probably best if there is some discussion around a solution. If my "PubSubLocker" seems like it is the solution I can open a PR here with it.
# Questions
While digging through the codebase I noticed that the `Subscription` type doesn't have the same "gotcha" as `Topic`'s do, in that you can only call `Join` once. As far as I can tell, it is perfectly valid for multiple different goroutines to call `Subscribe` on the same topic handler, however this will probably lead to wasted uses of resources, and I suspect lower throughput.
The one difference however is that there is no public `Subscribe` interface on the PubSub routers as there is with `Topic`. Currently my workaround is similar to how I handle topics in my previous `PubSubLocker`, but the issue is that if this pubsub router is passed to other services that don't have access to the `PubSubLocker`, they will be able to share a single `Subscription` interface.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing Topic Join and Subscription Subscribe APIs described in the issue, along with the PubSub router interfaces. The issue does not name files or tests; work is complete only after the project agrees on a topic and subscription management design and defines its expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100