allenai / allenai/olmo-eval

[BUG] vLLM venv mismatch between harness preset and model preset

Offen
#123 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
70
Forks
21
Ø Merge
2 T. 11 Std.
Gemergte PRs (30 T.)
41

Beschreibung

[Description (from Claude)]
# `vllm_isolated_venv` mismatch between harness preset and model preset provider kind

## Bug

When launching with `--harness default`, the job is correctly set up with `vllm_isolated_venv=True` because the `default` preset specifies `kind=ProviderKind.VLLM_SERVER` ([presets.py:74](src/olmo_eval/harness/presets.py#L74)). This causes vLLM to be installed in `/opt/vllm-venv` and stripped from the main `/opt/venv` during the install step ([launcher.py:704-720](src/olmo_eval/launch/beaker/launcher.py#L704-L720)).

However, `get_provider_extras` is called with the model spec rather than the harness provider kind ([job_assembler.py:416](src/olmo_eval/cli/beaker/job_assembler.py#L416)):

```python
provider_extras = get_provider_extras(exp.model_spec)
```

For model presets with `kind=ProviderKind.VLLM` (e.g. `olmo-3-1025-7b`), this correctly adds `"vllm"` to extras, which combined with `vllm_isolated_venv=True` triggers the isolated install. But at runtime, the model preset's own `kind=vllm` can take precedence, causing the runner to try to import vLLM from the main venv where it is no longer present.

## Root cause

`vllm_isolated_venv` is determined by the harness provider kind ([job_assembler.py:409](src/olmo_eval/cli/beaker/job_assembler.py#L409)):

```python
vllm_isolated_venv = provider_kind == "vllm_server"
```

But the runtime provider kind can diverge from the install-time assumption when the model preset specifies a different `kind` than the harness preset.

## Steps to reproduce

1. Use a model preset with `kind=vllm` (e.g. `olmo-3-1025-7b`)
2. Launch with `--harness default`
3. Job installs vLLM into `/opt/vllm-venv` only
4. At runtime the `vllm` provider is selected and fails to import from the main venv

## Expected behavior

The provider kind used at install time and runtime should be consistent. Either the harness preset's provider kind should fully override the model preset at runtime, or `vllm_isolated_venv` should only be set when the runtime provider is confirmed to be `vllm_server`.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.