open-compass / open-compass/AgentCompass

[Bug] pass@k and avg@k collapse to attempt-1 accuracy in micro-weighted aggregation

Open
#273 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
135
Forks
28
Avg merge
1d 10h
Merged PRs (30d)
37

Description

I noticed that pass@k and avg@k are reported incorrectly when the aggregation mode is micro_weighted.

The metric helpers first compute the correct overall pass@k and avg@k values. However, aggregate_all_metrics passes the same per_cat_counts to accuracy, pass@k, and avg@k. In micro-weighted mode, aggregate_with_policy ignores the metric-specific per-category values and always calculates correct / total from these counts, which represent attempt-1 accuracy.

Here is a minimal example:

from types import SimpleNamespace

from agentcompass.runtime.metrics.helpers import aggregate_binary_metrics

results = [
    {
        "task_id": "t1",
        "category": "coding",
        "attempts": {
            "1": {"correct": False},
            "2": {"correct": True},
        },
    },
    {
        "task_id": "t2",
        "category": "coding",
        "attempts": {
            "1": {"correct": False},
            "2": {"correct": True},
        },
    },
]

print(aggregate_binary_metrics(results).metrics)
print(
    aggregate_binary_metrics(
        results,
        SimpleNamespace(avgk=False),
    ).metrics
)

The expected values are:

accuracy = 0.0
avg_at_k = 0.5
pass_at_k = 1.0

The reported values are:

accuracy = 0.0
avg_at_k = 0.0
pass_at_k = 0.0

I reproduced this on revision 16cb375779271d4524cde785d1656e1fc4528fac with Python 3.11.9. No model or external API is required.

A possible fix would be to preserve the correctly computed overall values in micro-weighted mode, or provide separate numerators and denominators for each metric instead of reusing the attempt-1 accuracy counts.

I'd be happy to work on the fix and add focused regression coverage for avg@2 and pass@2. Please let me know if this approach looks reasonable.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in agentcompass.runtime.metrics.helpers at aggregate_binary_metrics, then trace aggregate_all_metrics and aggregate_with_policy to see how per-category counts are reused. Reproduce the supplied two-result example and add focused regression coverage for avg@2 and pass@2. Done means micro_weighted aggregation reports accuracy 0.0, avg_at_k 0.5, and pass_at_k 1.0.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
analytics
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.