KV get incremental update
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Feature Description
The feature is to provide a mechanism to return a diff (incremental update) of the KV store since a specified index.
#### Use Case(s)
Currently when you want to maintain a copy of the KV store (potentially filtered to a given prefix), and you perform a blocking read to monitor changes, when there is a change consul returns the entire KV store under the prefix. This can be problematic if the number of keys under the prefix is large, as this wastes bandwidth and processing power.
In our specific case, we have a primary Consul cluster in one datacenter, and then other Consul servers in datacenters across the globe. We then replicate a portion of the primary KV store to all the other Consul servers across the globe. However when there is any sort of update, this results in effectively dumping and sending the entire KV store, which contains thousands of keys, and when you're dumping it to locations on the other side of the world, it takes a while.
So if we had an incremental update feature, this could be used to only send the changes, and thus be much faster.
#### More description
The idea would be that the response would contain a list similar to the one that it does now, but filtered to only contain values with a ModifyIndex greater than the index specified in the API call. Deletions are also needed, but those can easily be represented by a null value, as otherwise entries always have a value, even if an empty string.
I looked into this, and it actually seems all the infrastructure is already in the code to do this. The behavior can be toggled by using the existing `filter` parameter, with a value of `changed`. The [kvsGraveyard](https://github.com/hashicorp/consul/blob/75bda848731b8fd6e49a2f8461f46af7e4d43866/agent/consul/state/state_store.go#L114) could be used to find the keys which have been deleted.
The problem is that the graveyard is kept pruned to only contain the last deletion, and nothing older. For this to work it'd need to be changed to hold a certain amount, either that be a fixed number, or expired after a certain age. Expiration age would probably be better, as with a fixed number if the caller is making a blocking read, they could end up waiting too long and have the graveyard expire out. With an age, we could ensure that a blocking read returns before the expiration time, providing an empty change list but with a new index for the caller to call again with.
Once we have a reasonable amount of entries in the graveyard, if a request comes in which the index has expired from the graveyard, then I can think of 2 possible behaviors. Either the call returns a 404, possibly with a message indicating that it's the index which isn't found. Or it can switch into return-everything mode, in which it behaves exactly as it does now, dumping the full store under the prefix. This latter behavior might seem a bit odd, but when you think about the use case, if the caller can't perform an incremental update, they're going to have to perform a full resync anyway. I would probably favor the 404 behavior though.
Contributor guide
Assessment
This issue has not been assessed yet.