Remove artificial 64 operations-per-transaction limit
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
Please search the existing issues for relevant feature requests, and use the reaction feature (https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to add upvotes to pre-existing requests.
#### Feature Description
Remove the limit of only 64 operations per transaction
#### Use Case(s)
## 1
We sometimes need to perform lots of operations against Consul's key/value store. For example one recent task required performing about 55k key updates. Performing these updates one by one was pegging the CPU, and would have taken approximately 12 hours to complete. Switching to using transactions (performing 64 key updates per transaction) dramatically sped this task up, and it completed in about 20 minutes. But even this was still pegging the CPU, resulting in lots of error messages about timeouts within the cluster and from other nodes.
While looking at the source code, it seems this limit of only being able to perform 64 operations per transaction is an artificial limit, and not due to any actual protocol limitations or anything. https://github.com/hashicorp/consul/blob/85b20d371396feecd310600b8c7d60aa1c532dbd/agent/txn_endpoint.go#L121-L126
Seeing as performing multiple updates per transaction resulted in a massive performance increase, it stands to reason that if we were able to perform more operations per transaction, the performance would increase even further, reducing the period in which the CPU is pegged and erroring. If we could perform the entire task in a single transaction, we may be able to avoid the issue entirely (just a theory). Therefore, since there is no technical reason this limit exists, it should be removed.
## 2
Due to the limitation of only 64 operations per transaction, we've engineered our solution to minimize potential issues by performing all inserts first, then updates, and then deletes. For the most part, this works. But it would be safer if we could perform the operations in a single transaction.
## 3
We have several hundred consul-template instances subscribing to key/value updates, and every time an update happens, it results in consul-templating rebuilding the output file. With 64 updates per transaction, this isn't too bad, but it does result in unnecessary reload operations (for the software which is consuming the templated files).
Contributor guide
Assessment
This issue has not been assessed yet.