allenai / allenai/asta-autodiscovery
run_mcts spins forever when the data-loader node can't be created, writing GBs of log
- Lenguaje dominante
- Python
- Estrellas
- 10
- Forks
- 2
- Merge medio
- 1 d 21 h
- PR fusionados (30 d)
- 11
Descripción
Found while smoke-testing the 1.0.0 release from PyPI. **Not a 1.0.0 regression** — the code is identical at 0.2.2 (`3e976267`). Any durable model failure at the data-loader step triggers it; #78 just made it easy to reach.
## Symptom
A run whose model calls fail persistently does not exit. It emits, at full speed with no backoff:
```
SAMPLED 1 NODE(S) FOR EXPANSION: ['0_0']
==========================
(1/1): EXPANDING NODE 0_0
No new experiment generated for node 0_0. Skipping this iteration.
```
Measured: **3.9 GB of log in 90 seconds** (~204M lines), of which exactly 3 lines were the underlying error. The real failure is unfindable, and an unattended run will fill the disk.
## Mechanism
[`mcts_utils.py:271`](../blob/v1.0.0/packages/autodiscovery/src/autodiscovery/mcts_utils.py#L271):
```python
# If the data loader node hasn't been executed, return the root. This is not run in batch mode.
if len(nodes_by_level[1]) == 0:
return [root]
```
The root's only job is to produce the data-loader node at level 1. If the model call that generates it fails, `nodes_by_level[1]` stays empty, so `select_nodes` returns `[root]` **unconditionally and forever**. This early return never consults [`has_untried_experiments()`](../blob/v1.0.0/packages/autodiscovery/src/autodiscovery/mcts.py#L194), so the `allow_generate_experiments = False` guard set at [`run.py:541`](../blob/v1.0.0/packages/autodiscovery/src/autodiscovery/run.py#L541) has no effect on this path.
Meanwhile the outer loop at [`run.py:490`](../blob/v1.0.0/packages/autodiscovery/src/autodiscovery/run.py#L490):
```python
while n_sampled < total_to_sample:
```
never advances, because the node is never committed — the failure path `return None`s at `run.py:546` without incrementing `n_sampled`. The `if not next_nodes: break` escape at `run.py:503` is unreachable, since `select_nodes` always returns a non-empty list. No iteration cap, no consecutive-failure counter, no sleep.
## Triggers
Anything that makes data-loader generation fail durably rather than transiently:
- misconfigured or missing Vertex project/location (#78)
- expired or insufficient credentials
- quota exhaustion, or a model the project has no access to
- a model whose responses never parse into an experiment
## Fix
Two independent guards, either of which stops the flood; both are cheap:
1. **Bound no-progress iterations in `run_mcts`.** Track consecutive iterations that commit no node and abort with the last underlying error once past a small threshold. This catches every cause, not just the data-loader case.
2. **Stop returning `[root]` unconditionally.** Once the root has failed to produce a data loader and has no untried experiments, `select_nodes` should return `[]` so the existing `if not next_nodes: break` fires.
(1) alone is sufficient and is the more general fix. Doing both means the loop terminates even if a future selection method reintroduces an unconditional return.
## Acceptance
- [ ] A run whose experiment generation always fails terminates with a non-zero exit and surfaces the underlying error, rather than looping.
- [ ] Log output for that run is bounded — no unbounded repetition of the same four lines.
- [ ] A regression test drives `run_mcts` with an experiment generator that always fails and asserts termination.
Related: #34's unchecked *"Enable public repo integration tests"* item — this is precisely the class of bug that only appears when a real model call fails durably, so no current test can catch it.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.