IntelLabs / IntelLabs/FloorSet
Evaluation wrapper ownership and the `__init__` / `solve()` timing boundary
- Dominant language
- Jupyter Notebook
- Stars
- 119
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
# Evaluation wrapper ownership and the `__init__` / `solve()` timing boundary
**Q1. Which `op_wrapper.py` is used during evaluation?**
A7 states that the organizers wrap the participant's executable with a Python script,
while A11 attaches an example `op_wrapper.py` and gives the command
`python iccad2026_evaluate.py --evaluate op_wrapper.py`.
Our submission ships its own `op_wrapper.py`, a `MyOptimizer(FloorplanOptimizer)` that
adheres to the `solve()` input/output schema of `optimizer_template.py` per A8/A9/A10.
Which of the following applies?
(a) the `op_wrapper.py` contained in the submission is used as-is, or
(b) the organizers replace it with their own wrapper.
The distinction is not cosmetic. The attached example wrapper launches the executable
once per case via `subprocess.run(..., timeout=60)`, so every case repeats the process
start, framework import and checkpoint load. We measure ~18 s per case that way, and the
first invocation on a cold machine exceeded the 60 s timeout.
**Q2. What may `__init__` contain?**
`iccad2026_evaluate.py` instantiates the optimizer once and times only `solve()`:
```python
optimizer = MyOptimizer() # not timed
start = time.time()
positions = optimizer.solve(...) # timed
runtime = time.time() - start
```
A14 states that library and checkpoint loading times are excluded from runtime.
Beyond that loading, our `__init__` runs a one-time warm-up on a synthetic 6-block dummy
case (uniform areas, two pins, no constraints). It exists to pay costs that would
otherwise land entirely on the first test case: CUDA context creation, cuDNN autotune,
`torch.compile` of the sampler, worker-process fork, and first-call initialization of the
LP and legalization libraries. That first sample costs 8.16 s against a 0.35 s steady
state.
No test-case data is read, cached or pre-solved during `__init__`. The warm-up input is
synthetic and fixed, and every timed `solve()` computes its result from its arguments.
Is a synthetic warm-up of this kind within the intended scope of initialization, or
should it be counted inside `solve()`?
Contributor guide
No contributing guide indexed for this repository
Research direction
Read iccad2026_evaluate.py, the submission op_wrapper.py, optimizer_template.py, and the referenced A7-A14 rules; compare the documented wrapper and timing boundaries. Run the stated python iccad2026_evaluate.py --evaluate op_wrapper.py command only to reproduce the described behavior. Done means the organizers clarify which wrapper runs and whether synthetic initialization warm-up is excluded from solve() timing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, pytorch
- Domain
- performance, tooling
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100