deeppavlov / deeppavlov/AutoIntent

Advisor treats `description_llm` / `description_typesafe` as embedder consumers and estimates no API cost for them

Open
#356 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
54
Forks
16
PR merge metrics
No merged PRs in 30d

Description

What happens

The advisor classifies description_llm and description_typesafe as embedder consumers: both are in _CACHE_HONORING_MODULES, _EMBEDDER_FORWARD_TRANSFORMER_MODULES and _EMBEDDER_CONSUMING_MODULES (_resource.py:99-117, :601-619). Neither loads an embedder — they call an external API — and the advisor has no notion of an API-cost driver, so the report is wrong in both directions.

autointent-advisor inspect zero-shot-typesafe --n-classes 77 --n-samples 300 --json on fa4df2d2 (zero-shot-typesafe is description_typesafe + the decision node only):

"resource": {"disk_download_gb": 0.0, "disk_cached_gb": 0.91, "ram_gb": 1.80, "vram_gb": 0.0,
             "time_hours": 0.0, "drivers": []}
Verdict: feasible (headroom: ample)
  • The 0.9 GB is the default sentence-transformers/all-MiniLM-L6-v2 ("already cached" here; "to download" on a fresh machine) and the RAM includes _embedder_load_ram_gb for it — a model the preset never loads.
  • drivers: [], time_hours: 0.0: nothing about the actual cost — the number of API calls (≈ unique utterances of train_1 + validation + test, × question types for typesafe, thanks to the answer cache), their price, or the wall-clock at max_concurrent/max_per_second. Measured in Darinochka/AutoIntent-experiments#43: description_llm on gpt-6 via OpenRouter ≈ $2.5–3.1 and ~10 s per 100 utterances; description_typesafe ≈ $0.01–0.02 per 100.
  • No preflight finding that the arm needs an API key in the environment (OPENAI_API_KEY / TYPESAFE_API_KEY), which is the thing that actually fails on a fresh machine.

Proposed

A third module class next to "embedder consumer" / "trainer": API scorer — no embedder, VRAM or model disk; an api driver row with the estimated call count (from the split sizes and, for typesafe, the number of question_type values in the search space), the configured rate limits → wall-clock, and cost: not estimated (external API, provider pricing) rendered explicitly rather than as $0/"ample"; plus a preflight check for the key env var.

Follow-up from #350.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/autointent/advisor/_estimates/_resource.py at the referenced module classifications and resource estimation logic, then run the zero-shot-typesafe inspect command described in the issue. Trace how split sizes, question types, rate limits, and preflight checks are represented. Done means API modules no longer load embedder resources, report API call and timing drivers, leave provider pricing unestimated, and check the required key.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.