deepmodeling / deepmodeling/LAMBench

[Code scan] Downstream aggregation drops domains instead of incomplete models

Open Beginner friendly
#438 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
20
Forks
11
Avg merge
30m
Merged PRs (30d)
4

Description

This issue was found by a Codex global repository scan of tracked non-test files at commit `8c93925cb10b401b2b83c738bd9263fd74474468`.

### Relevant code
https://github.com/deepmodeling/LAMBench/blob/8c93925cb10b401b2b83c738bd9263fd74474468/lambench/metrics/vishelper/metrics_calculations.py#L152-L164

### Impact
`domain_results` has models as rows and domains as columns, but `dropna(axis=1)` removes columns. If one model is missing a downstream domain, that domain is dropped for every model, and the incomplete model can still be ranked using the remaining domains.

This changes leaderboard scores by removing valid domains globally instead of excluding or penalizing the incomplete model.

### Suggested fix
Make missing-domain policy explicit. For example, compute domain means with `skipna=False` and then `dropna(axis=0)` for incomplete models, or apply a documented penalty to missing domains. Add a regression test with one model missing a single domain.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with lambench/metrics/vishelper/metrics_calculations.py at lines 152-164 and trace how domain_results is aggregated and filtered. Add the regression test described in the issue, using one model missing a single domain; done means incomplete models are handled by the chosen documented policy without dropping that domain globally.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.