allenai / allenai/tutormoments
Decide the fate of the deepseek-v4-pro arm: Together no longer serves it serverless
- Lenguaje dominante
- Python
- Estrellas
- 17
- Forks
- 3
- Merge medio
- 2 d 10 h
- PR fusionados (30 d)
- 12
Descripción
## Problem
The `deepseek-v4-pro` arm (model `deepseek-ai/DeepSeek-V4-Pro`) currently fails `tutormoments smoke`: Together returns
> 400 `model_not_available` — "Unable to access non-serverless model deepseek-ai/DeepSeek-V4-Pro. Please visit https://api.together.ai/models/deepseek-ai/DeepSeek-V4-Pro to create and start a new dedicated endpoint for the model."
i.e. the model moved off Together's serverless tier. Nothing in our wire format is at fault (this arm sends no thinking knobs at all); the arm is simply unrunnable on the current account setup. All other default-roster arms pass smoke.
## Decision needed (benchmark-defining)
This is a model-roster choice, so it needs an explicit owner decision rather than a code fix:
1. **Provision a dedicated Together endpoint** for DeepSeek-V4-Pro — keeps the arm identical, adds hourly endpoint cost and an ops dependency (endpoint must be running when a benchmark run starts).
2. **Swap to a serverless DeepSeek variant** — changes the benchmarked model; results would no longer be comparable to any prior V4-Pro runs.
3. **Drop/park the arm** — remove it from the default roster (or leave it with a documented "requires dedicated endpoint" note) until there's a reason to run it.
Whatever the choice, verify with `tutormoments smoke --arms deepseek-v4-pro` and update `default_config.yaml` / `models.yaml` accordingly.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Evaluación
Este issue todavía no se ha evaluado.