Upstream Catalyst load balancing

Upstreaming Catalyst Load Balancer

How it works right now

• There is a Serf cluster
• Every catalyst-api boots up a child MistUtilLoad process
• This MistUtilLoad process is fed up-to-date information about the current servers in the cluster
• Every five seconds, this MistUtilLoad process queries metrics from every Catalyst node in the cluster
• This data is cached locally, giving each node a global view of the current stream state
• When a user stream comes in, MistUtilLoad is queried to make intelligent decisions about playback routing

How it should work

• There is a Serf cluster
• Every catalyst-api boots up a child MistUtilLoad process
• This MistUtilLoad process is fed up-to-date information about the current servers in the cluster
• Every five seconds, this MistUtilLoad process queries metrics from every Catalyst node in the cluster pushes its current stream information into the Serf cluster; nodes forward this information so it reaches every node
• This data is cached locally, giving each node a global view of the current stream state
• When a user stream comes in, MistUtilLoad is queried to make intelligent decisions about playback routing are made based on this cached information

Start by locating the existing Serf integration, catalyst-api startup path, and MistUtilLoad process handling described in the issue. Trace how stream information is currently collected and routed; done means nodes publish current stream state through the Serf cluster, cache the global view locally, and route playback without the child MistUtilLoad process.