huggingface / huggingface/cadgenbench
Editing fixtures ship BRepCheck-invalid input.step (202, 217, 240) — penalises the no-op baseline
- Dominant language
- Python
- Stars
- 125
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Several **editing** fixtures provide a starting `input.step` that is **itself invalid** under the project's own validity gate (`sanity_check_submission.py` / `analyze_step`). Confirmed:
| Fixture | `input.step` gate | Defect |
|---|---|---|
| **202** | `is_valid=False, watertight=False` | BRepCheck not well-formed |
| **217** | `is_valid=False, watertight=False` | `BRepCheck_UnorientableShape` (1 face) |
| **240** | `is_valid=False` | `BRepCheck_UnorientableShape` — 1 invalid face of 1004 |
(Found while running the GPT-5.5 + build123d-mcp system over the full set; 202/217/240 were the editing failures, and all three trace to a broken *input*, not the agent's edit.)
## Why this matters
1. **It zeroes the documented baseline.** `docs/benchmark/submission.md` says the unchanged import is a valid, scored fallback, and the editing metric renormalises the shape axis against the **no-op input**. But if the input fails the validity gate, the no-op result also fails the gate → `cad_score = 0`. So an honest "return the input unchanged" submission scores **zero** on these fixtures, and the edit-vs-baseline renormalisation is computed against a zero baseline.
2. **It silently requires healing a broken input.** To score anything, a system must first *repair the supplied geometry* before it can even apply the requested edit — which is not what the editing task is meant to measure. 217's defect is clearable by a capable agent; **240's is not** — it resisted ShapeFix, sewing, defeaturing, drop-sew, and local-cut (we built and adversarially tested a dedicated `recover()` heal tool and could not fix 240 without distorting the part). So 240 is effectively **unscoreable above zero** for any honest system.
3. **It distorts cross-system comparison.** Two systems that both correctly apply the edit can differ purely on whether they happened to heal the inherited input defect — noise unrelated to the editing capability being measured.
## Suggested fixes (any of)
- **Regenerate** these `input.step` files so they pass the validity gate (preferred — they presumably *should* be valid starting solids).
- Or **exclude** 240 (and any input that can't be made valid) from the scored set, or mark it so the no-op baseline isn't penalised.
- Add a **CI/data check** that runs `sanity_check_submission.py` over every editing `input.step` so an invalid starting solid can't ship.
## Reproduce
```bash
DATA=$(python -c 'from cadgenbench.common.paths import data_inputs_dir; print(data_inputs_dir())')
for f in 202 217 240; do echo "== $f =="; python "$DATA/sanity_check_submission.py" "$DATA/$f/input.step"; done
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running data_inputs_dir()/sanity_check_submission.py against fixtures 202, 217, and 240, then read analyze_step and docs/benchmark/submission.md to confirm the validity and baseline requirements. Check the affected input.step files and the editing-fixture data path; done means the affected inputs pass the validity gate, or the scoring and CI behavior explicitly handles any excluded fixture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100