Automattic / Automattic/wp-codebox

agent task examples need a half-configured provider-stack regression guard

Open
#1,044 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
16
Forks
4
Avg merge
59m
Merged PRs (30d)
131

Description

## Problem

WP Codebox examples and runtime contract correctly show that provider stacks need explicit top-level fields on the agent-task input: `provider`, `model`, `provider_plugin_paths`, and `secret_env`. In practice, orchestrators repeatedly regress into a half-configured state where they pass `provider_plugin_paths` but omit `provider` and/or `secret_env`.

That state is confusing because the provider plugin gets mounted, but provider selection/credential exposure is not guaranteed by WP Codebox. The failure then appears later inside the embedded runtime as a provider/OAuth failure rather than as a clear contract problem at recipe/request construction time.

Today's WPSG typed-artifact Lab loop hit this exact shape: provider plugin path present, but no top-level `provider`, `model`, or Codex `secret_env`. The runtime then reached Data Machine/Codex and failed with `Codex OAuth refresh failed.` Regenerating the plan with the full WP Codebox contract made Homeboy fail fast correctly on missing runner secret refs before runtime execution.

## Existing good contract

The cookbook example already shows the right shape in `examples/recipes/cookbook/codex-agent-smoke.json`:

- provider plugin mounted in `inputs.extra_plugins`
- `inputs.secretEnv` includes the five `AI_PROVIDER_OPENAI_CODEX_*` names
- workflow args include `provider=codex` and `model=gpt-5.5`

The runtime-core builder also has the right top-level fields: `provider`, `model`, `provider_plugin_paths`, and `secret_env`.

## Expected

Add an explicit regression guard/diagnostic around half-configured provider stacks so orchestrators get a clear signal before this reaches an embedded runtime.

Possible acceptable fixes:

- Add a smoke test that builds an agent-task recipe from input with `provider_plugin_paths` but no `provider`/`secret_env` and asserts the output/diagnostic makes the missing contract obvious.
- Add a non-secret diagnostic/warning in recipe/task metadata when `provider_plugin_paths` is non-empty but provider or secret env names are absent.
- Strengthen docs/tests around the cookbook example so provider plugin paths are never treated as provider selection or credential exposure.

This should stay generic: WP Codebox should not hardcode Codex-specific behavior outside the cookbook example. The generic lesson is that mounting a provider plugin is not the same as selecting a provider or exposing credentials.

## Acceptance criteria

- Automated coverage prevents a provider-plugin-only agent-task input from looking fully configured.
- The diagnostic or test names the missing generic fields: `provider` and `secret_env`.
- No secret values are accepted or logged.
- Do not run Cargo locally for this fix; WP Codebox verification should use its normal Node/TypeScript tests or CI/Lab as appropriate.

## Related context

This is a recurring integration regression between orchestrators and WP Codebox. The Homeboy/Homeboy Extensions side is tracked separately; this issue is for WP Codebox examples/contract guardrails so future callers do not repeat the same half-configured provider stack.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with examples/recipes/cookbook/codex-agent-smoke.json and the runtime-core agent-task builder described in the issue. Run the normal Node/TypeScript tests, then add coverage for input containing provider_plugin_paths without provider or secret_env. Done means the diagnostic or assertion names both missing generic fields, does not expose secret values, and keeps provider behavior generic.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.