allenai / allenai/olmo-eval

[BUG] vLLM venv mismatch between harness preset and model preset

Aberta
#123 0 comentários 0 reações 0 responsáveis Ver no GitHub
bug
Linguagem predominante
Python
Estrelas
70
Forks
21
Merge médio
2d 11h
PRs com merge (30d)
41

Descrição

[Description (from Claude)]
# `vllm_isolated_venv` mismatch between harness preset and model preset provider kind

## Bug

When launching with `--harness default`, the job is correctly set up with `vllm_isolated_venv=True` because the `default` preset specifies `kind=ProviderKind.VLLM_SERVER` ([presets.py:74](src/olmo_eval/harness/presets.py#L74)). This causes vLLM to be installed in `/opt/vllm-venv` and stripped from the main `/opt/venv` during the install step ([launcher.py:704-720](src/olmo_eval/launch/beaker/launcher.py#L704-L720)).

However, `get_provider_extras` is called with the model spec rather than the harness provider kind ([job_assembler.py:416](src/olmo_eval/cli/beaker/job_assembler.py#L416)):

```python
provider_extras = get_provider_extras(exp.model_spec)
```

For model presets with `kind=ProviderKind.VLLM` (e.g. `olmo-3-1025-7b`), this correctly adds `"vllm"` to extras, which combined with `vllm_isolated_venv=True` triggers the isolated install. But at runtime, the model preset's own `kind=vllm` can take precedence, causing the runner to try to import vLLM from the main venv where it is no longer present.

## Root cause

`vllm_isolated_venv` is determined by the harness provider kind ([job_assembler.py:409](src/olmo_eval/cli/beaker/job_assembler.py#L409)):

```python
vllm_isolated_venv = provider_kind == "vllm_server"
```

But the runtime provider kind can diverge from the install-time assumption when the model preset specifies a different `kind` than the harness preset.

## Steps to reproduce

1. Use a model preset with `kind=vllm` (e.g. `olmo-3-1025-7b`)
2. Launch with `--harness default`
3. Job installs vLLM into `/opt/vllm-venv` only
4. At runtime the `vllm` provider is selected and fails to import from the main venv

## Expected behavior

The provider kind used at install time and runtime should be consistent. Either the harness preset's provider kind should fully override the model preset at runtime, or `vllm_isolated_venv` should only be set when the runtime provider is confirmed to be `vllm_server`.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.