feature: add river inspector that returns various info about the system

Motivation

I want to periodically upload metrics about jobs count, state, and kind.

The API that allows that would be great, but at the same time I have a workaround and I think this is a low prio issue.

API

I think about something similar as asynq inspector, but only with few methods like:

• JobsCountByState
• JobsCountByKind
• etc...

This API might be utilized by riverui, because right now riverui uses its own sql queries

API dosen't need to guarantee backward compatibility as it will probably change in upcoming versions.

Possible solutions

1. River middleware that sends telemetry data about each job.

This would work, but many projects enable sampling to reduce the cost of storing large amount of data.

2. subscriptions

This approach requires a leader election to make sure only one instance sends data.
Btw - I think exposing a leader election as an API would be a great value added by river.

3. Collector

This is the most common setup and collector could aggregate metrics. But if you want to keep things simple then instances send the telemetry data straight to the external service.

4. Perioding job + sql

Right now I use periodic job and custom sql queries to fetch the data and send it once a while.

Start by reviewing River's subscriptions documentation and the asynq Inspector API for the proposed inspection shape. Compare those options with riverui's internal/dbsqlc SQL queries and the listed approaches for collecting job metrics. Done would require an agreed API for counts by state and kind, but the issue does not define the final scope.