allenai / allenai/tutormoments

Decide the fate of the deepseek-v4-pro arm: Together no longer serves it serverless

Abierto
#53 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
17
Forks
3
Merge medio
2 d 10 h
PR fusionados (30 d)
12

Descripción

## Problem

The `deepseek-v4-pro` arm (model `deepseek-ai/DeepSeek-V4-Pro`) currently fails `tutormoments smoke`: Together returns

> 400 `model_not_available` — "Unable to access non-serverless model deepseek-ai/DeepSeek-V4-Pro. Please visit https://api.together.ai/models/deepseek-ai/DeepSeek-V4-Pro to create and start a new dedicated endpoint for the model."

i.e. the model moved off Together's serverless tier. Nothing in our wire format is at fault (this arm sends no thinking knobs at all); the arm is simply unrunnable on the current account setup. All other default-roster arms pass smoke.

## Decision needed (benchmark-defining)

This is a model-roster choice, so it needs an explicit owner decision rather than a code fix:

1. **Provision a dedicated Together endpoint** for DeepSeek-V4-Pro — keeps the arm identical, adds hourly endpoint cost and an ops dependency (endpoint must be running when a benchmark run starts).
2. **Swap to a serverless DeepSeek variant** — changes the benchmarked model; results would no longer be comparable to any prior V4-Pro runs.
3. **Drop/park the arm** — remove it from the default roster (or leave it with a documented "requires dedicated endpoint" note) until there's a reason to run it.

Whatever the choice, verify with `tutormoments smoke --arms deepseek-v4-pro` and update `default_config.yaml` / `models.yaml` accordingly.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.