IntelLabs / IntelLabs/FloorSet

Evaluation wrapper ownership and the `__init__` / `solve()` timing boundary

Open
#15 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
119
Forks
35
PR merge metrics
No merged PRs in 30d

Description

# Evaluation wrapper ownership and the `__init__` / `solve()` timing boundary

**Q1. Which `op_wrapper.py` is used during evaluation?**

A7 states that the organizers wrap the participant's executable with a Python script,
while A11 attaches an example `op_wrapper.py` and gives the command
`python iccad2026_evaluate.py --evaluate op_wrapper.py`.

Our submission ships its own `op_wrapper.py`, a `MyOptimizer(FloorplanOptimizer)` that
adheres to the `solve()` input/output schema of `optimizer_template.py` per A8/A9/A10.
Which of the following applies?

(a) the `op_wrapper.py` contained in the submission is used as-is, or
(b) the organizers replace it with their own wrapper.

The distinction is not cosmetic. The attached example wrapper launches the executable
once per case via `subprocess.run(..., timeout=60)`, so every case repeats the process
start, framework import and checkpoint load. We measure ~18 s per case that way, and the
first invocation on a cold machine exceeded the 60 s timeout.

**Q2. What may `__init__` contain?**

`iccad2026_evaluate.py` instantiates the optimizer once and times only `solve()`:

```python
optimizer = MyOptimizer() # not timed
start = time.time()
positions = optimizer.solve(...) # timed
runtime = time.time() - start
```

A14 states that library and checkpoint loading times are excluded from runtime.

Beyond that loading, our `__init__` runs a one-time warm-up on a synthetic 6-block dummy
case (uniform areas, two pins, no constraints). It exists to pay costs that would
otherwise land entirely on the first test case: CUDA context creation, cuDNN autotune,
`torch.compile` of the sampler, worker-process fork, and first-call initialization of the
LP and legalization libraries. That first sample costs 8.16 s against a 0.35 s steady
state.

No test-case data is read, cached or pre-solved during `__init__`. The warm-up input is
synthetic and fixed, and every timed `solve()` computes its result from its arguments.

Is a synthetic warm-up of this kind within the intended scope of initialization, or
should it be counted inside `solve()`?

Contributor guide

No contributing guide indexed for this repository

Research direction

Read iccad2026_evaluate.py, the submission op_wrapper.py, optimizer_template.py, and the referenced A7-A14 rules; compare the documented wrapper and timing boundaries. Run the stated python iccad2026_evaluate.py --evaluate op_wrapper.py command only to reproduce the described behavior. Done means the organizers clarify which wrapper runs and whether synthetic initialization warm-up is excluded from solve() timing.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, pytorch
Domain
performance, tooling
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.