Proposal: Service/Task Stats

Re: https://github.com/docker/docker/issues/24597

I was doing some thinking about this problem and wanted to propose a possible solution to solicit feedback before I started working on it. The current problem is that there is no easy way to get stats for a service and/or task. You can currently get task stats by running docker stats on the container id, but you have to be on that node.

Saving stats to managers:

1. The agent/worker will get a new reporter for stats.
   • The worker will keep track of subscribing/unsubscribing from container stats (existing logic in daemon) based on the current listen method.
   • All stats from the worker will be sent to managers via GRPC stream UpdateStats(stream obj). Stats will be the same data we push on that endpoint currently but protobuf, but it will include the task id + service id for db indexes. That way we get both task & service stats depending on how you query.
2. The manager will listen for stat updates via grpc simliar to the dispatcher w/ task status updates.
   • When an update is received it will overwrite the stat value from before for the given task. This means only the last value will ever be stored in raft. When a task gets removed the stat will get removed for that task.
   • Updates should be done batchy just like task status updates.
   • A new table will need to be made task_stats with an index on service id & task id to support service/task queries.

Getting stats:

• add a new rpc method GetStats(StatRequest) stream StatsResponse where you can query by either service id or task id. Querying by both is the same as just task id so should be fine also. StatsResponse will contain a list of stats ( 1 per task ). Technically this could be a bidirectional stream since you could have multiple requests for any number of tasks. Then you could just use the queue/sink stuff from go-events to push out. The client would push listen/remove requests & the server would push responses back.

Questions:

• Does this seem ok 👼 ?
• Should this be added to the dispatcher service? or a new service? The latter seems to make more sense to me, but not sure.
• Is collecting stats every 1 second OK? That's the current hardcoded value in the daemon for the stats collector.
• Should the StatRequest support multiple tasks/services? I can see tasks, but displaying multiple services might get interesting

Doing things this way seems to enable easily adding support for autoscaling in the future since the managers will have access to the aggregated stats for a service. Then you can watch for updates -> run against service rules -> schedule updates accordingly.

Start by reviewing the existing daemon stats collector, the dispatcher’s task status updates, and the gRPC service definitions mentioned in the proposal. Done would require an agreed design for collection, manager storage, querying, and service ownership before implementation can be scoped.