bytedance / bytedance/PXMeter

Benchmark integrity check misses seed-incomplete match keys

Open
#16 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
100
Forks
12
PR merge metrics
No merged PRs in 30d

Description

## Description

Benchmark integrity checking groups rows by `entry_id`, but downstream intersection and aggregation operate on `(entry_id, chain_id_1, chain_id_2)` match keys. An entry can therefore pass the seed/sample completeness check even though each individual chain/interface key is incomplete.

## Minimal reproduction

Use one entry with two match keys and two expected seeds:

```text
match_key seed sample
(E, A, B) 1 0
(E, C, D) 2 0
```

Run:

```python
_integrity_check(df, seeds_to_check=["1", "2"])
```

Expected: the data is reported incomplete because neither match key has both seeds.

Actual at current `main` (`72117242154d0c9f52ce727dc352210aab76d495`):

```text
set()
```

The entry-level group sees two distinct `(seed, sample)` values and treats the entry as complete.

## Root cause

`_integrity_check()` groups `_seed_sample` counts by `entry_id`. `_prepare_tasks()` subsequently forms intersections with `match_key` and later metrics are aggregated at chain/interface granularity.

## Impact

Seed-asymmetric chain or interface rows can enter the reported intersection. Models may then be compared on different effective observations, undermining reproducibility and bootstrap/statistical comparisons.

## Why this issue does not include a PR yet

The mechanical grouping change is small, but the correct policy needs confirmation: drop only incomplete match keys, drop the whole entry when any key is incomplete, or require a dataset-specific expected key set. These choices change benchmark cohorts and published result comparability. A safe fix should be validated on representative multi-seed benchmark outputs, which are not included in the repository and are too large to infer from the unit fixture alone.

## Environment

- Python 3.11.15
- pandas 3.0.5
- PXMeter current `main`: `72117242154d0c9f52ce727dc352210aab76d495`
- Reproduced with a two-row in-memory DataFrame

Contributor guide

Open the contributing guide

Research direction

Start by reading _integrity_check() and _prepare_tasks(), then run the two-row in-memory reproduction described in the issue. Confirm the intended policy for incomplete match keys before changing the grouping logic. Done means the chosen policy is implemented and validated against representative multi-seed benchmark outputs, not only the unit fixture.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.