openai / openai/codex

[Windows][plugin-eval] Absolute paths become C:\C:\..., causing 19/28 evaluator test failures

Open
#44,090 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug CLI skills windows-os
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of Codex CLI is running?

codex-cli 0.153.4

What platform are you using?

Windows 10 x64

What issue are you seeing?

The official plugin-eval@openai-curated-remote plugin (Plugin Eval 0.1.2) has a Windows path-normalization failure where an already-absolute Windows path can be prefixed again and become malformed, for example:

C:\C:\Users\...

The Plugin Eval executable itself can run successfully on Windows, but the full evaluator test suite is heavily affected by this path bug.

Reproduction
  1. Install/enable plugin-eval@openai-curated-remote on Windows.
  2. Run a minimal standalone evaluation fixture.
  3. Observe that the fixture succeeds (100/100, exit code 0).
  4. Run the full Plugin Eval test suite.
  5. Observe failures involving malformed absolute Windows paths with a duplicated drive prefix such as:
C:\C:\Users\...
Actual result

Before updating Codex CLI (0.146.1):

9 passed
19 failed

After updating Codex CLI to stable 0.153.4:

9 passed
19 failed

The failure pattern is unchanged, suggesting this is in Plugin Eval path handling rather than the Codex CLI version.

A simple minimal-skill fixture evaluates successfully at 100/100, so this is not a total Windows incompatibility.

Expected behavior

Absolute Windows paths such as:

C:\Users\<USER>\...

should remain absolute and should never be joined or prefixed into:

C:\C:\Users\...

The evaluator test suite should normalize and resolve native Windows paths correctly.

Impact
  • 19/28 Plugin Eval tests fail on Windows.
  • Scores for real local skills cannot be treated as fully authoritative while path-related tests are failing.
  • Reliable regression testing of Codex skills/plugins on native Windows is blocked.
Additional information
  • Codex install method: global npm @openai/codex
  • Plugin: plugin-eval@openai-curated-remote
  • Plugin Eval version: 0.1.2
  • No Plugin Eval source files, manifests, symlinks, or local workarounds were modified.
  • A local skill was evaluated statically only; none of its operational capabilities were executed.

The Plugin Eval source currently lives in openai/plugins, but Issues are disabled in that repository, so this report is filed here because the affected plugin is distributed through the official Codex plugin catalog.

If useful, I can provide a sanitized list of the 19 failing test names/path traces without including the local Windows username.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the full Plugin Eval test suite on native Windows and compare it with the successful minimal standalone fixture. Inspect the Plugin Eval source in openai/plugins for path normalization and resolution, then verify that all 28 tests handle native absolute paths without producing duplicated drive prefixes such as C:\C:\Users....

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, rust
Domain
cli, operating-systems, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.