apple / apple/pfl-research

Track per-client metrics over time

Open
#91 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
358
Forks
43
Avg merge
9h 2m
Merged PRs (30d)
2

Description

tracking the metrics of each user over time over the course of training the global model can be very useful for distribution of metrics, monitoring outlier users and debugging.
The user metric should be measured at a central iteration only if it actually was sampled of course.
We can have a post processor (https://apple.github.io/pfl-research/reference/postprocessor.html#pfl.postprocessor.base.Postprocessor) that dumps the metrics to disk for offline analysis (a postprocessor have access to an individual user's metrics).
The offline part to analyze and visualize the per-client metrics over time is outside the scope for this GH issue.
This solution must be compatible with distributed simulations. this may require an all-gather if multi-node simulations, but being restricted to single node multi-gpu simulations for this feature is OK.
The result should be (csv?) file(s) with per-client metrics.

Contributor guide

Open the contributing guide

Research direction

Start with the referenced pfl.postprocessor.base.Postprocessor documentation and trace how an individual user's metrics are exposed during central iterations. Define how sampled metrics are collected and emitted as per-client CSV file(s), then verify the approach works for distributed simulations or explicitly supports single-node multi-GPU simulations. Done means metrics are recorded only when the client was sampled and are available for offline analysis.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.