[BUG] vLLM venv mismatch between harness preset and model preset
- Vorherrschende Sprache
- Python
- Sterne
- 70
- Forks
- 21
- Ø Merge
- 2 T. 11 Std.
- Gemergte PRs (30 T.)
- 41
Beschreibung
[Description (from Claude)]
# `vllm_isolated_venv` mismatch between harness preset and model preset provider kind
## Bug
When launching with `--harness default`, the job is correctly set up with `vllm_isolated_venv=True` because the `default` preset specifies `kind=ProviderKind.VLLM_SERVER` ([presets.py:74](src/olmo_eval/harness/presets.py#L74)). This causes vLLM to be installed in `/opt/vllm-venv` and stripped from the main `/opt/venv` during the install step ([launcher.py:704-720](src/olmo_eval/launch/beaker/launcher.py#L704-L720)).
However, `get_provider_extras` is called with the model spec rather than the harness provider kind ([job_assembler.py:416](src/olmo_eval/cli/beaker/job_assembler.py#L416)):
```python
provider_extras = get_provider_extras(exp.model_spec)
```
For model presets with `kind=ProviderKind.VLLM` (e.g. `olmo-3-1025-7b`), this correctly adds `"vllm"` to extras, which combined with `vllm_isolated_venv=True` triggers the isolated install. But at runtime, the model preset's own `kind=vllm` can take precedence, causing the runner to try to import vLLM from the main venv where it is no longer present.
## Root cause
`vllm_isolated_venv` is determined by the harness provider kind ([job_assembler.py:409](src/olmo_eval/cli/beaker/job_assembler.py#L409)):
```python
vllm_isolated_venv = provider_kind == "vllm_server"
```
But the runtime provider kind can diverge from the install-time assumption when the model preset specifies a different `kind` than the harness preset.
## Steps to reproduce
1. Use a model preset with `kind=vllm` (e.g. `olmo-3-1025-7b`)
2. Launch with `--harness default`
3. Job installs vLLM into `/opt/vllm-venv` only
4. At runtime the `vllm` provider is selected and fails to import from the main venv
## Expected behavior
The provider kind used at install time and runtime should be consistent. Either the harness preset's provider kind should fully override the model preset at runtime, or `vllm_isolated_venv` should only be set when the runtime provider is confirmed to be `vllm_server`.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.