ContextLab / ContextLab/orchestrator

HuggingFace Inference API provider

Open
#484 1 comment 0 reactions 0 assignees View on GitHub
architecture enhancement
Dominant language
Python
Stars
3
Forks
2
Avg merge
13m
Merged PRs (30d)
1

Description

## Context

The provider set is Dartmouth Chat + HuggingFace only (decided 2026-08-21; ADR 0001). HuggingFace support means the **hosted Inference API**, not the local transformers inference in `integrations/huggingface_model.py` / `lazy_huggingface_model.py` — those stay in the frozen layer and retire under #430.

## What to build

A `HuggingFaceProvider` in the canonical layer (`models/providers/`), speaking the Inference API over HTTP with `aiohttp` (already a core dependency — same shape as `DartmouthProvider`, no new core deps, no provider extra required unless the SDK is used).

- Text-generation against a small set of served models, selected from the live catalog rather than a hard-coded list (two rounds of hard-coded Anthropic ids were wrong; do not repeat that here).
- `HF_TOKEN` from the environment or `~/.orchestrator/.env`, same as other credentials.
- Free-tier vs paid distinction must fail closed: a model the token cannot serve must raise, not silently route elsewhere. Model fallback across providers does not exist (ADR 0001).
- Registered in the normal model registry when a credential is present, so pipelines can select the models without touching the provider directly.

## Acceptance criteria

- [ ] Contract tests (hermetic, recorded/mocked HTTP boundary is fine at this layer) covering: request shape, auth header, error mapping (401/403/402/429/5xx), model-unavailable, and the registry integration.
- [ ] A `live-huggingface` job in `live-tests.yml`, mirroring `live-dartmouth`: hard-fails when `HF_TOKEN` is missing under `ORCHESTRATOR_REQUIRE_LIVE=1`, and fails when every test skips.
- [ ] Live tests exercise at least one real served model end-to-end: `generate()` returns non-empty content, `models.list()`/discovery works, `health_check()` works.
- [ ] Reasoning-model behaviour handled: a response that spends its token budget on reasoning and returns empty content must not be reported as a successful empty answer (learned from the Dartmouth work — see #432 comments).
- [ ] README provider table flips HuggingFace to **supported** only after the live job passes remotely.

## Out of scope

- Local transformers/torch inference (frozen; retires with #430).
- `text-generation-inference` self-hosting, Inference Endpoints (dedicated), and any SDK dependency in core.

Related: #429 (adapter-layer consolidation — this provider lands in `models/providers/`), #430 (retirement of the frozen layer), ADR 0001 provider policy.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.