mgechev / mgechev/skillgrade

Feature: configure custom LLM providers for the built-in Codex agent

Open
#31 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
711
Forks
46
PR merge metrics
No merged PRs in 30d

Description

## Problem

Skillgrade supports custom OpenAI-compatible endpoints through `OPENAI_BASE_URL` and `OPENAI_MODEL`, but the current documentation describes these settings as applying to LLM grading and `skillgrade init`.

They do not configure the built-in `agent: codex` execution path.

The Codex agent is invoked with a fixed command and Skillgrade does not expose a supported way to provide the Codex CLI with:

- a custom model provider;
- a provider-specific base URL;
- a provider-specific model;
- the API-key environment variable expected by that provider; or
- a provider-specific Codex configuration/profile inside Docker trials.

This makes it difficult to use Skillgrade with OpenAI-compatible gateways or other Codex-supported custom providers while retaining the built-in Codex agent and Docker isolation.

## Reproduction

Given an evaluation such as:

```yaml
version: "1"

defaults:
agent: codex
provider: docker
trials: 1
```

and a skill-directory `.env` containing placeholder configuration such as:

```dotenv
CUSTOM_PROVIDER_API_KEY=
OPENAI_BASE_URL=https://example.invalid/v1
OPENAI_MODEL=custom-model
```

Skillgrade passes environment variables into the trial, but does not generate or provision a Codex configuration that selects the custom provider or maps the provider's API key.

The Codex agent still runs through the fixed built-in invocation:

```text
codex exec --full-auto --skip-git-repo-check --json --ephemeral
```

As a result, the custom endpoint/provider configuration is not reliably used by the Codex agent.

## Suggested fix

Please consider adding an explicit, backwards-compatible configuration surface for the built-in Codex agent.

The exact schema can follow the Codex CLI configuration model, but it should support the equivalent of:

- provider name;
- base URL;
- model;
- API-key environment variable name;
- optional Codex configuration/profile content or path; and
- separate local and Docker handling.

For example, a future configuration could conceptually support:

```yaml
defaults:
agent: codex
provider: docker

codex:
model_provider: custom
model: custom-model
model_providers:
custom:
base_url: https://example.invalid/v1
env_key: CUSTOM_PROVIDER_API_KEY
```

The implementation should not require API-key values to be stored in `eval.yaml`. Secrets should continue to come from the environment or an ignored `.env` file, and should not be persisted in reports or session logs.

For Docker trials, Skillgrade could either:

1. generate an isolated Codex configuration inside the trial; or
2. accept a repository-local, non-secret Codex configuration template and explicitly inject only the selected secret environment variables.

The existing default behavior for `agent: codex` should remain unchanged when no custom configuration is provided.

## Acceptance criteria

- A Codex evaluation can select a custom provider, endpoint, and model.
- Provider-specific API keys can be supplied through environment variables without being committed or written to `eval.yaml`.
- The configuration works consistently with both `provider: local` and `provider: docker`, subject to the expected isolation differences.
- The selected provider and model are visible in sanitized diagnostic output.
- API-key values and authentication caches are never written to reports, session logs, Docker images, or committed files.
- Existing Codex evaluations without custom configuration continue to work.
- The documentation clearly distinguishes:
- Skillgrade LLM-grader provider settings;
- Codex-agent provider settings; and
- credentials required by the invoked Codex CLI.

## Current workaround

The current workaround is to use the `command` agent with a custom wrapper, or maintain a small downstream patch that provisions the required Codex configuration. A first-class Codex configuration surface would avoid requiring each consumer to maintain its own integration.

## Environment

- Skillgrade: 0.2.2
- Agent: codex
- Provider: docker
- Runtime: Node.js 20+
- Desired use case: OpenAI-compatible custom LLM provider

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace how eval.yaml and .env settings reach the fixed `codex exec` invocation and how environment variables are handled for local and Docker trials. Compare the existing LLM-grader settings with the Codex path, then define configuration and sanitized diagnostics that satisfy the acceptance criteria without exposing secrets while preserving the current default behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, node.js, typescript
Domain
cli, devops, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.