huggingface / huggingface/cadgenbench

Editing fixtures ship BRepCheck-invalid input.step (202, 217, 240) — penalises the no-op baseline

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
125
Forks
12
PR merge metrics
No merged PRs in 30d

Description

## Summary

Several **editing** fixtures provide a starting `input.step` that is **itself invalid** under the project's own validity gate (`sanity_check_submission.py` / `analyze_step`). Confirmed:

| Fixture | `input.step` gate | Defect |
|---|---|---|
| **202** | `is_valid=False, watertight=False` | BRepCheck not well-formed |
| **217** | `is_valid=False, watertight=False` | `BRepCheck_UnorientableShape` (1 face) |
| **240** | `is_valid=False` | `BRepCheck_UnorientableShape` — 1 invalid face of 1004 |

(Found while running the GPT-5.5 + build123d-mcp system over the full set; 202/217/240 were the editing failures, and all three trace to a broken *input*, not the agent's edit.)

## Why this matters

1. **It zeroes the documented baseline.** `docs/benchmark/submission.md` says the unchanged import is a valid, scored fallback, and the editing metric renormalises the shape axis against the **no-op input**. But if the input fails the validity gate, the no-op result also fails the gate → `cad_score = 0`. So an honest "return the input unchanged" submission scores **zero** on these fixtures, and the edit-vs-baseline renormalisation is computed against a zero baseline.

2. **It silently requires healing a broken input.** To score anything, a system must first *repair the supplied geometry* before it can even apply the requested edit — which is not what the editing task is meant to measure. 217's defect is clearable by a capable agent; **240's is not** — it resisted ShapeFix, sewing, defeaturing, drop-sew, and local-cut (we built and adversarially tested a dedicated `recover()` heal tool and could not fix 240 without distorting the part). So 240 is effectively **unscoreable above zero** for any honest system.

3. **It distorts cross-system comparison.** Two systems that both correctly apply the edit can differ purely on whether they happened to heal the inherited input defect — noise unrelated to the editing capability being measured.

## Suggested fixes (any of)

- **Regenerate** these `input.step` files so they pass the validity gate (preferred — they presumably *should* be valid starting solids).
- Or **exclude** 240 (and any input that can't be made valid) from the scored set, or mark it so the no-op baseline isn't penalised.
- Add a **CI/data check** that runs `sanity_check_submission.py` over every editing `input.step` so an invalid starting solid can't ship.

## Reproduce

```bash
DATA=$(python -c 'from cadgenbench.common.paths import data_inputs_dir; print(data_inputs_dir())')
for f in 202 217 240; do echo "== $f =="; python "$DATA/sanity_check_submission.py" "$DATA/$f/input.step"; done
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running data_inputs_dir()/sanity_check_submission.py against fixtures 202, 217, and 240, then read analyze_step and docs/benchmark/submission.md to confirm the validity and baseline requirements. Check the affected input.step files and the editing-fixture data path; done means the affected inputs pass the validity gate, or the scoring and CI behavior explicitly handles any excluded fixture.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.