actions-metrics-server should report the current status of the workers

What would you like added?

Our team adopted the metrics that were added via https://github.com/actions/actions-runner-controller/pull/2057, however, they did not provide the insight we were hoping for. We are trying to better understand the current state of our workers, and understanding why a given job may be stuck in the queue. The metrics from the PR provide some helpful insight (specifically the rate at which jobs are created and completed), but they do not give us a high-level view of which workers are busy, and with what.

My suggestion is to add these new metrics, both of which are gauges, and both have the normal commonLabels:

• github_workflow_jobs_running: Currently running jobs. Does not include Pending jobs, only ones that have started on a worker.
• github_workflow_jobs_queued: Jobs waiting for a worker. If there are any additional labels we can provide to explain why the job is queued (ie, HRA limit exceeded, pod is pending, etc) that would be helpful. But that extra data is not required.

Open to name changes and label changes on the above suggestions.

Why is this needed?

When we have many repos using github actions runners, it is hard to diagnose why a given workflow is stuck pending. We want to know which jobs are using up our runners so we can focus on optimizing them, segregating them, or giving them more resources. Currently, we have no automatic way to gather this data.

Additional context

This is a suggested improvement to the work that came from https://github.com/actions/actions-runner-controller/pull/2057.

Start at the actions-metrics-server entry point and read the metrics work introduced by PR #2057. Define the intended semantics and labels for gauges covering running and queued jobs, including whether queue-reason data is feasible. Done means the metrics expose current worker status and their behavior is covered by the project's existing metrics validation.