Event Stream configurable event durability
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
https://github.com/hashicorp/nomad/pull/9147 removed event durability from the event stream. The initial implementation attempted to acquire a go-memdb transaction outside of the FSM causing other valid FSM transactions from locking up the state store.
### Background
Event durability was meant to allow for servers to maintain a configurable buffer of events in our state store so that the event buffer used for the event stream could survive restarts. When a server came back online it could fill rehydrate the in memory event buffer with what was persisted on disk.
When a given transaction was committed, events would be generated and inserted into the events table. Insertions with this approach were fine, unfortunately maintaining the table size is where we encountered problems and ultimately decided to remove the functionality until we could come up with a better alternative.
Pruning the event table mutated state store state outside of the FSM which caused transaction locking to occur. The in memory event broker had an evict callback which allowed us to call into the state store to delete the event from go-memdb, which required a write transaction outside of the FSM, which contended with other valid FSM applies https://github.com/hashicorp/nomad/pull/9147/files#diff-53c7d524897a192ece819816a9dcf006dee46747fbb853e7f638601ea556a72cL129
When we re-visit event durability in the future it will be necessary to add new events and evict old events all within the same transaction, which occurs on the fast path of the FSM. A fixed-table size in go-memdb would be really nice but we are thinking other approaches as well.
### Plan Forward
To enable a fixed amount of durable events to be persisted the state store, the state store on each server will likely need an atomic counter to keep track of the amount of events stored within the events go-memdb table. This way we can evict and insert a single entry when events are created and not require a transaction outside of the FSM.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.