mpfaffenberger / mpfaffenberger/code_puppy

config.py model resolution guesses on failure: _validate_model_exists caches True on exceptions; vision default falls back to hardcoded 'gpt-4.1' that may not exist

Open
#415 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
814
Forks
278
Avg merge
2d 5h
Merged PRs (30d)
76

Description

Problem

Two related robustness issues in code_puppy/config.py's model resolution, both "in the face of ambiguity, guess" violations:

1. _validate_model_exists() caches and returns True on any failure (lines ~491-512)
    except Exception:
        # If we can't validate, assume it exists to avoid breaking things
        _model_validation_cache[model_name] = True
        return True

A transient failure in ModelFactory.load_config() (corrupted extra_models.json, plugin exception) makes every model name "valid" — and the wrong answer is then cached for the process lifetime, so even after the underlying problem resolves, get_global_model_name() keeps returning a stored model that may not exist, and the user gets a confusing downstream ValueError: Model 'X' not found in configuration from the factory instead of the intended graceful fallback. At minimum don't cache the failure result; better, log the exception.

2. _default_vision_model_from_models_json() hardcodes "gpt-4.1" fallback (lines ~450-489)

When no model config is loadable it returns the literal \"gpt-4.1\" — a model that is not in the user's catalog (the bundled models.json is 4 bytes / empty by design now). Callers will then fail looking up a phantom model. The non-vision sibling _default_model_from_models_json() was already fixed to return None and warn (per its docstring "Returning None lets callers detect the no-model state"); the vision variant should do the same instead of guessing a name. Also the preferred_candidates tuple (claude-4-0-sonnet, gemini-2.5-flash-preview-05-20) is a stale hardcoded heuristic that drifts from the actual catalog.

Suggested fix

    except Exception:
        logger.warning(\"Model validation failed for %s\", model_name, exc_info=True)
        return True  # permissive, but do NOT cache

and make _default_vision_model_from_models_json() return Optional[str], mirroring _default_model_from_models_json() (return None + one-shot warning), letting callers handle absence explicitly.

Filed by Zen Reviewer A (code-puppy-60635a)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in code_puppy/config.py with _validate_model_exists() and _default_vision_model_from_models_json(), then inspect their callers and the matching non-vision default helper. Verify that validation failures are not cached and that vision resolution can signal no available model without selecting a hardcoded name; done means the fallback behavior is explicit and avoids phantom catalog entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.