allenai / allenai/asta-autodiscovery

Vertex AI: VERTEX_LOCATION no longer defaults to `global`, and a missing project fails silently

Open
#78 2 comments 0 reactions 1 assignee Claimed by @gas2own View on GitHub
enhancement
Dominant language
Python
Stars
10
Forks
2
Avg merge
1d 21h
Merged PRs (30d)
11

Description

Found while smoke-testing the 1.0.0 release from PyPI. Both are regressions introduced by #68, and the second one means the **published** docs describe behavior the package doesn't have.

## Problem

[`llm.py:339`](../blob/v1.0.0/packages/autodiscovery/src/autodiscovery/llm.py#L339) passes Vertex settings to litellm only when the env var happens to be set:

```python
def _provider_kwargs(model: str) -> dict[str, Any]:
if provider_of(model) != VERTEX_AI:
return {}
kwargs: dict[str, Any] = {}
if project := os.getenv(VERTEX_PROJECT_ENV_VAR):
kwargs["vertex_project"] = project
if location := os.getenv(VERTEX_LOCATION_ENV_VAR):
kwargs["vertex_location"] = location
return kwargs
```

### 1. `VERTEX_LOCATION` unset no longer means `global`

0.2.2's `vertex_config.get_vertex_openai_base_url()` did:

```python
location = os.getenv(VERTEX_LOCATION_ENV_VAR) or "global"
```

Now, unset means the key is simply absent and litellm applies its own default of `us-central1` — where the default model `vertex_ai/gemini-3.7-flash` is not served. The result is a 404 that reads like the model doesn't exist:

```
litellm.NotFoundError: Vertex_aiException - {
"error": {
"code": 404,
"message": "Publisher model `projects//locations/us-central1/publishers/google/models/gemini-3.7-flash` was not found or your project does not have access to it."
```

Both `packages/autodiscovery/RELEASE.md` (the PyPI long description) and `docs/autodiscovery/standalone.md` still say `VERTEX_LOCATION` is *"optional; defaults to global"*. As published, that is not true.

### 2. `VERTEX_PROJECT_ID` unset fails silently

0.2.2 raised immediately and named the variable:

> Vertex AI configuration is required for Gemini models. Set `VERTEX_OPENAI_BASE_URL` or both a project ID (`VERTEX_PROJECT_ID`) and an optional location (`VERTEX_LOCATION`, defaults to global).

Now nothing validates it. litellm falls back to whatever project Application Default Credentials happens to carry, so the 404 above names a project the operator never selected. Anyone whose ADC quota project differs from their Vertex project gets a confusing error pointing at the wrong place.

## Why it matters more than it looks

These are the first two env vars every new user sets, and the defaults are Vertex models, so this is the most likely first-run failure. It also compounds badly: because the failure is a durable per-call error rather than a startup error, it triggers the unbounded loop in 79, which buries the 404 under gigabytes of log output.

## Repro

With `VERTEX_PROJECT_ID` and `VERTEX_LOCATION` unset but valid ADC present:

```sh
auto-discovery --name test --description "some csv" \
--n_experiments 4 --out_dir ./out data.csv
```

Expected: a startup error naming the missing variable. Actual: a 404 about a model in `us-central1` under an unrelated project.

## Fix

Restore both behaviors in `_provider_kwargs`, or better, validate in `run.resolve_model_args` where every other model precondition is already checked:

- Default `vertex_location` to `global` when `VERTEX_LOCATION` is unset, matching the documented contract.
- Raise `ModelError` naming `VERTEX_PROJECT_ID` when a `vertex_ai/` model is selected and no project is configured, before the first model call.

## Acceptance

- [ ] `VERTEX_LOCATION` unset resolves to `global`, and a test covers it.
- [ ] A `vertex_ai/` model with no `VERTEX_PROJECT_ID` fails at startup with a message naming the variable.
- [ ] `RELEASE.md` and `standalone.md` match actual behavior.
- [ ] `ModelError` is caught in `easy.cli_main` and exits with the message rather than a ~20-line traceback. Today every model-flag mistake — including the `/` prefix error that all 0.2.x users will hit on upgrade — surfaces as what looks like a crash.

Related: #54 (which required env vars depend on which provider/backend is active is exactly this class of problem, though scoped there to the webstack rather than the CLI's model flags). Touches the same module as #70.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.