hashicorp / hashicorp/consul

config entry replication should use hash based comparisons

Open
#9,196 0 comments 1 reaction 0 assignees View on GitHub
theme/performance type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

Currently replication of config entries to secondary datacenters behaves differently than how ACL tokens, policies, and roles replicate (and how intentions replicated in versions of Consul prior to 1.9.0).

On startup a secondary will request ALL config entries from the primary. It will then do a differential comparison with what is stored already in the secondary:

- for missing entries, they are inserted
- for extra entries, they are deleted
- for existing entries, **they are updated blindly**

After the first round of replication, we use blocking queries and the last seen primary raft index to avoid write churn in the secondaries for the 3rd bullet point above. This means that every time there is a new election in a secondary datacenter, all of the config entries are transferred and re-written to the local state store, which is not ideal on both the network transfer and write churn fronts.

Compare this to ACL item replication:

- for missing entries, they are inserted
- for extra entries, they are deleted
- for existing entries, they are updated if the hashes of the content don't match

The initial fetch also does not include the full body of the item, but instead only key envelope metadata and the hash of the full contents. This cuts down on network transfer to get started.

As we move more stuff over to config entries, it would be beneficial to switch config entries to replicate in the more efficient manner.

Note: do not accidentally introduce a version of this bug again: https://github.com/hashicorp/consul/pull/9022

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.