Provide: announce different CID sets to different routing systems (HTTP vs DHT), and select the target router per `provide once` / `routing provide` call
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
### Checklist
- [x] This is a feature request, not a question.
- [x] I have searched the issue tracker.
- [x] I am running the latest kubo version.
### Version
```
Kubo version: 0.42.0
```
### Description
There is currently no way to announce one set of CIDs to HTTP routers and a different, smaller set to the Amino DHT. Three things combine to make it impossible:
1. **`Provide.Strategy` is global.** It produces a single key stream, and `routing/composer.go` has a single `ProvideRouter` used by both `Composer.Provide` and `Composer.ProvideMany`. Whatever `Routing.Methods.provide` points at receives everything.
2. **`ipfs provide once` / `/api/v0/routing/provide` cannot target a routing system.** The only option is `--recursive`, and `nd.Provider.ProvideOnce(mh)` has no notion of a destination. Applications that announce explicitly, rather than relying on the reprovide cycle, cannot express the split either.
3. **The IPNS-over-pubsub rendezvous key shares that same single `ProvideRouter`.** `TopicDiscovery()` builds `routing.NewRoutingDiscovery(ContentRouting)`, so advertising a topic is an ordinary `ContentRouting.Provide(CID(sha256("floodsub:")))`. There is therefore no way to keep a pubsub topic discoverable in the DHT while keeping content CIDs out of it, which is precisely the case where we most want DHT reach.
### Motivation
I maintain [pkc-js](https://github.com/pkcprotocol/pkc-js), a pubsub-based social protocol built on kubo. Each community node publishes an IPNS record, subscribes to a pubsub topic for that community, and stores a large and constantly growing set of content blocks whose root rotates on every update.
My intention is simple: **announce to the DHT only the few CIDs related to a community's IPNS and its pubsub swarm**, specifically the IPNS-over-pubsub rendezvous CID for the topic and the current root, and let the HTTP routers carry everything else. Once a reader has found the swarm for a community, the peers in that topic mesh serve its blocks over bitswap, and our clients also query delegated HTTP routers, so per-block DHT provider records are not what makes content retrievable in our system. They are just the expensive part of the announce path, and they scale with the whole pinset rather than with the handful of keys that actually need global reach.
Today the only expressible options are "everything to both" or "everything to one".
### No workaround exists today
I could not find any way to express this with current config or CLI:
- `Routing.Methods.provide` accepts exactly one router, shared by the reprovide cycle and by topic advertisement.
- `Provide.Strategy` filters which CIDs are announced, but applies identically to every routing system behind that one router.
- `Provide.Enabled=false` disables the provide system wholesale, and takes application-driven announcements down with it: `ipfs provide once` fails outright with `cannot provide: Provide.Enabled is false`, and the deprecated `ipfs routing provide` silently succeeds while doing nothing, since `nd.Provider` is a `NoopProvider`.
- `ipfs provide once` has no router or system selector.
So with a mixed DHT plus HTTP configuration there is no supported way to get a single CID into the DHT without dragging the entire strategy set along with it.
### Proposed solutions
Either would work; (B) alone is sufficient for my use case, since pkc-js announces explicitly rather than relying on the reprovide cycle.
**(A) Per-routing-system provide strategy.** The `Provide` docs already state the section is "designed to support multiple routing systems in the future", so this fits the existing schema:
```json
{
"Provide": {
"Strategy": "all",
"DHT": { "Strategy": "roots" },
"HTTP": { "Strategy": "all" }
}
}
```
with `Provide.Strategy` remaining the default for any system that does not override it.
**(B) Per-call router selection.** A selector on the imperative announce path:
```console
$ ipfs provide once --routing-system=http
$ ipfs provide once --routing-system=dht
```
or, under `Routing.Type=custom`, naming an entry from `Routing.Routers`:
```console
$ ipfs provide once --routers=HttpRoutersParallel
```
with a matching query parameter on `/api/v0/provide/once` and `/api/v0/routing/provide` so that HTTP RPC clients can pass it through. On the JS side this would surface as a field on `RoutingProvideOptions` in `kubo-rpc-client`, which today carries only `recursive`.
### Notes and relation to existing issues
- On the supported routing path this is currently moot, because `constructDefaultHTTPRouters` hardcodes `ProvideRouter: noopRouter` ("we don't have spec for sending provides to /routing/v1"), and with `Routing.Type=auto` the sweeping provider is DHT-only. HTTP providing is therefore only reachable via `Routing.Type=custom`, which the docs mark experimental and not for production. I understand that gates prioritisation, and that the real unblocker is IPIP-526 (ipfs/specs#526), where I have commented with this use case.
- Related: #11089 (HTTP router provides regressed, fixed in #11112), #11123 (`ipfs routing provide` fails for pubsub routing keys).
- Question for maintainers: once IPIP-526 lands, is there an intended way to express "announce to delegated routers, not to the DHT" on the supported `Routing.Type=auto` plus `Routing.DelegatedRouters` path, or is `Routing.Type=custom` expected to remain the only place this is possible?
Contributor guide
Research direction
Start with routing/composer.go and trace the shared ProvideRouter through Composer.Provide, Composer.ProvideMany, and the provide once CLI and /api/v0/provide/once entry points; also inspect constructDefaultHTTPRouters and RoutingProvideOptions. Done means the supported routing paths can announce distinct CID sets or target a selected router per call without changing unrelated provide behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, cli, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100