Benchmark integrity check misses seed-incomplete match keys
- Dominant language
- Python
- Stars
- 100
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
## Description
Benchmark integrity checking groups rows by `entry_id`, but downstream intersection and aggregation operate on `(entry_id, chain_id_1, chain_id_2)` match keys. An entry can therefore pass the seed/sample completeness check even though each individual chain/interface key is incomplete.
## Minimal reproduction
Use one entry with two match keys and two expected seeds:
```text
match_key seed sample
(E, A, B) 1 0
(E, C, D) 2 0
```
Run:
```python
_integrity_check(df, seeds_to_check=["1", "2"])
```
Expected: the data is reported incomplete because neither match key has both seeds.
Actual at current `main` (`72117242154d0c9f52ce727dc352210aab76d495`):
```text
set()
```
The entry-level group sees two distinct `(seed, sample)` values and treats the entry as complete.
## Root cause
`_integrity_check()` groups `_seed_sample` counts by `entry_id`. `_prepare_tasks()` subsequently forms intersections with `match_key` and later metrics are aggregated at chain/interface granularity.
## Impact
Seed-asymmetric chain or interface rows can enter the reported intersection. Models may then be compared on different effective observations, undermining reproducibility and bootstrap/statistical comparisons.
## Why this issue does not include a PR yet
The mechanical grouping change is small, but the correct policy needs confirmation: drop only incomplete match keys, drop the whole entry when any key is incomplete, or require a dataset-specific expected key set. These choices change benchmark cohorts and published result comparability. A safe fix should be validated on representative multi-seed benchmark outputs, which are not included in the repository and are too large to infer from the unit fixture alone.
## Environment
- Python 3.11.15
- pandas 3.0.5
- PXMeter current `main`: `72117242154d0c9f52ce727dc352210aab76d495`
- Reproduced with a two-row in-memory DataFrame
Contributor guide
Research direction
Start by reading _integrity_check() and _prepare_tasks(), then run the two-row in-memory reproduction described in the issue. Confirm the intended policy for incomplete match keys before changing the grouping logic. Done means the chosen policy is implemented and validated against representative multi-seed benchmark outputs, not only the unit fixture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100