stackstorm statsd metrics per action, by action_ref

Hey all!

Over here we've got a lot of production workflows running through stackstorm, but the builtin statsd metrics are rather thin. I see in the documentation, we get overall number of a given execution, by name, but nothing more for each status of said executions, by name.

This means we lack success/failure rate metrics to readily consume/visualize/address on any of our deployments.

I am not sure if this was intentional due to limiting statsd resources or potential contention from said metrics possibly exploding (obviously would mean for 1000x actions, you'd have potentially ~4-5k metrics just around statuses)

I would like to propose adding in to the codebase an OPTIONAL configuration setting for [metrics] that would essentially tell the application to generate status-per-action metrics at the user's behest, defaulting to the current state of baseline metrics. This would allow users to either adopt the newer, more granular statsd metrics OR continue with the default behavior.

I believe the adds would live within this section of the liveaction status update(s): https://github.com/StackStorm/st2/blob/master/st2common/st2common/util/action_db.py#L207-L312

Please let me know if you have any concerns about adding this stuff in here or not. I can analyze/add/etc just want to elicit some feedback/thoughts. THANK YOU!!!

Start in st2common/st2common/util/action_db.py, especially the liveaction status update section around lines 207-312, and trace how the existing StatsD metrics are emitted. Define the optional [metrics] setting and verify that status-per-action metrics are emitted only when enabled while baseline metrics remain unchanged.