open-compass / open-compass/AgentCompass
[Bug] pass@k and avg@k collapse to attempt-1 accuracy in micro-weighted aggregation
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 135
- Forks
- 28
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 37
Description
I noticed that pass@k and avg@k are reported incorrectly when the aggregation mode is micro_weighted.
The metric helpers first compute the correct overall pass@k and avg@k values. However, aggregate_all_metrics passes the same per_cat_counts to accuracy, pass@k, and avg@k. In micro-weighted mode, aggregate_with_policy ignores the metric-specific per-category values and always calculates correct / total from these counts, which represent attempt-1 accuracy.
Here is a minimal example:
from types import SimpleNamespace
from agentcompass.runtime.metrics.helpers import aggregate_binary_metrics
results = [
{
"task_id": "t1",
"category": "coding",
"attempts": {
"1": {"correct": False},
"2": {"correct": True},
},
},
{
"task_id": "t2",
"category": "coding",
"attempts": {
"1": {"correct": False},
"2": {"correct": True},
},
},
]
print(aggregate_binary_metrics(results).metrics)
print(
aggregate_binary_metrics(
results,
SimpleNamespace(avgk=False),
).metrics
)
The expected values are:
accuracy = 0.0
avg_at_k = 0.5
pass_at_k = 1.0
The reported values are:
accuracy = 0.0
avg_at_k = 0.0
pass_at_k = 0.0
I reproduced this on revision 16cb375779271d4524cde785d1656e1fc4528fac with Python 3.11.9. No model or external API is required.
A possible fix would be to preserve the correctly computed overall values in micro-weighted mode, or provide separate numerators and denominators for each metric instead of reusing the attempt-1 accuracy counts.
I'd be happy to work on the fix and add focused regression coverage for avg@2 and pass@2. Please let me know if this approach looks reasonable.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in agentcompass.runtime.metrics.helpers at aggregate_binary_metrics, then trace aggregate_all_metrics and aggregate_with_policy to see how per-category counts are reused. Reproduce the supplied two-result example and add focused regression coverage for avg@2 and pass@2. Done means micro_weighted aggregation reports accuracy 0.0, avg_at_k 0.5, and pass_at_k 1.0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- analytics
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100