Feature request: capability isolation + routing telemetry for reproducible plugin/skill/app testing

What feature would you like to see?

A first-class developer test surface for ChatGPT/Codex integrations that combines:

1. an enforceable capability allowlist/denylist for the current test run; and
2. privacy-bounded routing/invocation telemetry showing which capability was eligible, selected, invoked, and actually supplied data used by the result.

This is broader than ChatGPT Projects. Projects are only one place where the limitation is easy to reproduce.

Problem

When testing a plugin, skill, app, connector, MCP integration, or memory provider, a developer can observe a correct final answer without being able to prove which mechanism produced it.

For example:

retain -> fresh-session retrieval -> downstream action

A later correct answer may come from:

• native ChatGPT memory;
• project chat history;
• another installed app/plugin;
• another skill;
• a connector/context provider;
• Session Search or another host retrieval mechanism;
• or the provider actually under test.

Explicitly prompting use provider X is not a valid substitute because it changes the router input and stops measuring automatic selection.

A developer therefore needs to distinguish states such as:

capability unavailable
capability available but ineligible
capability eligible but not selected
capability selected but not invoked
capability invoked but returned no useful result
capability returned a result but another source won
capability result actually supplied the answer

Today these can collapse into one plausible user-visible response.

This is not a request for chain-of-thought. Event-level execution provenance is sufficient.

Requested capability 1: enforceable test isolation

Provide a narrow execution profile / test mode / run configuration where a developer can control the eligible capability set.

Example:

Memory / context providers
  Provider-under-test     ON
  Native project memory   OFF
  Other memory provider   OFF

Apps / connectors
  Google Drive            OFF
  GitHub                  OFF

Skills
  skill-under-test        ON
  unrelated-skill         OFF

The important property is enforcement: disabled capabilities must be ineligible for that run, not merely discouraged by prompt instructions.

The UI need not be project-specific. A task-level, session-level, developer profile, CLI flag, or test harness configuration would solve the core problem more generally.

Requested capability 2: routing / invocation telemetry

Expose a compact execution receipt such as:

capability discovered: yes/no
capability eligible: yes/no (+ reason when safe)
capability selected: yes/no/unknown
provider/tool action invoked: provider + action
result: success/failure/no-result/cancelled
retrieval source class: native memory / project chat / app / connector / skill / web / other
provider transaction/receipt id: when available
result contributed to final answer: observed / not observed / unknown

This could be available in developer mode, local logs, structured events, or an exported receipt.

Requested capability 3: disposable / resettable test runs

For integrations with persistent state, provide one of:

• disposable test namespaces;
• per-run canaries / isolated memory namespaces;
• deterministic reset semantics;
• or a documented cleanup contract.

Without this, a previous successful retain/write can make a later run PASS even if the current write never happened.

Why existing observability issues are related but insufficient

Related Codex issues already show parts of the same problem:

• #35869 — plugin Skill injection can occur while the documented telemetry event is missing;
• #41760 — request for first-class attributable SkillInvocation events;
• #39166 — skill-only plugin invocation differs across Codex surfaces.

Those are important, but invocation telemetry alone still leaves causal testing ambiguous when several competing capabilities remain simultaneously eligible.

The missing combination is:

controlled capability set
        +
observable routing/invocation
        +
clean/disposable state
        =
reproducible integration test

Reproducible research case

This gap surfaced while testing a ChatGPT memory provider. The test requires proving:

retain
  -> fresh-session retrieval
  -> downstream application

A clean Project reduces history contamination, but project-only memory does not itself isolate external apps/plugins/skills, and native project memory can still be a competing source. Project Instructions can request a routing policy but are model instructions, not an enforceable capability boundary.

The resulting methodology therefore has to mark hard provider isolation as BLOCKED and provider attribution as UNKNOWN / CONFOUNDED unless retrieval-time evidence identifies the provider.

Public research/evidence packet:
https://github.com/TeaShaman-cyber/theseus-memory-provider-lab/issues/7

Methodology PR:
https://github.com/TeaShaman-cyber/theseus-memory-provider-lab/pull/6

User impact

This is useful beyond developers:

• users can create intentionally narrow, high-trust execution environments;
• why did ChatGPT/Codex use this source/tool? becomes answerable without exposing hidden reasoning;
• project/task behavior becomes reproducible instead of depending on ambient account state;
• unrelated connected systems can be excluded from sensitive or focused work.

Cross-surface requirement

Where a plugin/skill/app can run in ChatGPT, Work, Codex Desktop, Codex CLI, or another supported surface, expose whether it was:

• available;
• enabled;
• eligible;
• selected;
• invoked;
• blocked by policy;
• or unsupported on that surface.

This avoids inferring runtime provisioning from final model behavior.

Separate but related feedback-loop issue

This request is about testing and observability.

A separate upstream feature request tracks the second developer-loop gap: submitting developer feedback itself should not require launching a general-purpose browser agent solely to operate a forum UI:

• #44798 — agent-native developer feedback integration without requiring Browser/Work

Acceptance sketch

A developer can run two otherwise-identical tests:

Run A: only Provider A eligible
Run B: only Provider B eligible

Each run produces an auditable receipt showing the effective capability set, routing/invocation events, and the provider/source that supplied the successful retrieval.

No chain-of-thought exposure is required.

No Codex files, tests, or entry points are named in the issue. Start by reviewing related issues #35869, #41760, and #39166 plus the acceptance sketch; the work is done when isolated runs produce auditable capability, routing, invocation, and result-contribution receipts.