hashicorp / hashicorp/nomad

Event Stream configurable event durability

Open
#9,565 1 comment 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/events type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

https://github.com/hashicorp/nomad/pull/9147 removed event durability from the event stream. The initial implementation attempted to acquire a go-memdb transaction outside of the FSM causing other valid FSM transactions from locking up the state store.

### Background

Event durability was meant to allow for servers to maintain a configurable buffer of events in our state store so that the event buffer used for the event stream could survive restarts. When a server came back online it could fill rehydrate the in memory event buffer with what was persisted on disk.

When a given transaction was committed, events would be generated and inserted into the events table. Insertions with this approach were fine, unfortunately maintaining the table size is where we encountered problems and ultimately decided to remove the functionality until we could come up with a better alternative.

Pruning the event table mutated state store state outside of the FSM which caused transaction locking to occur. The in memory event broker had an evict callback which allowed us to call into the state store to delete the event from go-memdb, which required a write transaction outside of the FSM, which contended with other valid FSM applies https://github.com/hashicorp/nomad/pull/9147/files#diff-53c7d524897a192ece819816a9dcf006dee46747fbb853e7f638601ea556a72cL129

When we re-visit event durability in the future it will be necessary to add new events and evict old events all within the same transaction, which occurs on the fast path of the FSM. A fixed-table size in go-memdb would be really nice but we are thinking other approaches as well.

### Plan Forward

To enable a fixed amount of durable events to be persisted the state store, the state store on each server will likely need an atomic counter to keep track of the amount of events stored within the events go-memdb table. This way we can evict and insert a single entry when events are created and not require a transaction outside of the FSM.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.