anthropics / anthropics/claude-cookbooks

[PROPOSAL] Cookbook: Measuring each sub-agent's real contribution to a result

Open
#723 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
52.7k
Forks
6.3k
Avg merge
25m
Merged PRs (30d)
6

Description

When several sub-agents (or several steps) contribute to one result, it is surprisingly hard to say how much each one actually mattered. The naive answers are wrong in opposite directions: splitting credit equally ignores that some steps did the real work, and crediting whoever produced the final text ignores everyone whose output it was built on. Getting this right matters for anything that routes reward, budget, or trust by contribution: which sub-agent to keep, which tool earned its cost, which step to cut.

There is a clean way to measure it borrowed from cooperative game theory. A contributor's marginal value is how much the result gets worse when you remove them, averaged over the orders in which contributors could have been added. The leave-one-out version (remove one, measure the drop) is the cheap approximation; the Shapley version (average the marginal drop across orderings) is the fairer one when contributions interact.

What the notebook would cover:
- A small multi-agent task with a measurable final score, where it is obvious by construction which sub-agent did the heavy lifting and which rode along.
- Leave-one-out credit first: re-run the task with each sub-agent removed, attribute the score drop. Show where it is fine and where it misleads (two agents that each cover for the other both look removable).
- Shapley credit on the same task: average the marginal contribution across orderings, and show it correctly separates the load-bearing agent from the passenger.
- The honest cost: exact Shapley is exponential in the number of contributors, so the notebook should show the sampling approximation and how many samples are enough for a stable ranking.
- The boundary worth stating: this measures contribution to a *measured* outcome. It is only as honest as the score function, so a weak or gameable score makes the attribution weak too.

It runs on the messages API with a small agent harness. I would target the agents section. This is the measurement piece under the cost and routing proposals already open — once you can attribute contribution, you can price and route by it.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.