streaming: support gracefully reconnecting a subscription
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
If a subscription is force closed by a server (because the server is being shutdown, or because of a state restore) the client will receive an error instructing them to reset their state and re-subscribe. If the subscription is closed because the server was shutdown, the client may be able to resume the subscription, but it will need to know that the state store was not restored to a different state while it was disconnected.
Clients will only be able to resume their streams if the Server is being being shutdown for a restart or upgrade. All of the clients will need to reconnect to other Servers to restart their streams. If the clients need to generate a fresh snapshot, this could create a lot of load on the remaining servers. Where as if they are able to resume, they won't need to request the initial large snapshot.
To handle this I believe we will need some unique identifier for the state that is shared by all servers. When the client re-subscribes it can send that identifier, and the server can either accept the subscription, or tell the client to clear its state and start again.
Ref: [original PR comment](https://github.com/hashicorp/consul/pull/8323/files/5e41040d3a6a54f1f00c1b96fa27694fd43471fc#diff-17e502d6563a0ebfdb62040a995eb946)
Contributor guide
Research direction
Start by reviewing the original PR comment linked in the issue and the shutdown, restore, and subscription-reconnect behavior it describes. Define how a shared state identifier is exchanged and validated so resumed streams avoid a fresh snapshot only when state is compatible; otherwise, clients must clear state and resubscribe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100