As a user, I can see task queue waiting time and service time metrics

There is a desire to scale Pulp to achieve an idea of a Service Operating Level (SOL). In Pulp for example, this would be a goal like: "95% of all tasks are started within X second". Or "99% of all sync tasks are started within X second". There is a desire with the operator for example to provide autonomic scaling based on the task queue.

The first step to doing that is to expose metrics about the waiting and service times of tasks in Pulp.

Terms

Waiting Time - The amount of time from a task being created until a task is Running/Canceled.
Service Time - The amount of time from a task started Running until a task is Finished/Canceled/Errored.

Proposal

{
	"waiting": {
		"99th_percentile": 12.18763,
		"95th_percentile": 9.1726,
		"average": 4.18172
	},
	"running": {
		"99th_percentile": 12.18763,
		"95th_percentile": 9.1726,
		"average": 4.18172
	}
}

Where to put this data?

We could put this into the status API. I think as is written ^ that would make sense.

That being said, I think it should get a new API endpoint. The issue is that there are likely two questions users want to answer which likely mean they want some GET style filtering to get at:

1. What is the time window over which this is summarized?
2. What are the metrics for tasks of type X? For example, maybe we care a lot about sync task waiting times and not others.

What's the time window?

The time window is really important because traffic for at least two reaons:

1. Traffic coming into Pulp is likely highly time varying. For example, say a cron job submits 200 sync's at 2am and the rest of the day there are no sync tasks. You want to make scaling decisions I think on recent data, not "all data".

2. Another important reason is an auto-scaler could waaaay overscale. Say you ran your Pulp for a long time without autoscaling and then you enable autoscaling. Having waiting time of "all history" would be difficult to statistically change because there is so much old data it would take a lot of "quick tasks" to actually move that number. Meanwhile the auto-scaler keeps provisioning, probably endlessly until eventually the CFO finds out.

Time scale GET params

I propose we include optional GET params in the first implementation because they are so important.

from = unix timestamp of the minimum time tasks should be filtered by for statistical computation. Optional.
to = unix timestamp of the maximum time tasks should be filtered by for statistical computation. Optional.

Start by reviewing the status API and the task lifecycle described in the issue, including when waiting and service time begin and end. Compare that with the proposed new endpoint and its optional from/to filters, then define how task-type filtering and the time window should work. Done means the API exposes waiting and running-time statistics with a clear scope and documented response.