1jehuang / 1jehuang/jcode

5 pre-existing test failures on master (env var isolation + stale assertions)

Open
#978 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

autonomous: no bug tech-debt triage: needs-decision
Dominant language
Rust
Stars
19.9k
Forks
2.3k
Avg merge
2d 7h
Merged PRs (30d)
30

Description

Summary

5 tests fail on the current master branch (c4cdc6768) when run in an environment with provider profile and OpenRouter env vars set. These are environment isolation issues in the test code, not bugs in the application code.

Failing tests

1. run_auto_poke_followup_targets_below_threshold_todos

Root cause: Stale assertion. The test asserts message.contains("completion confidence") but the message format was changed to use "Validate further:" instead. The assertion was never updated.

2. explicit_anthropic_api_choice_pins_api_key_over_available_oauth

Root cause: The test sets JCODE_HOME to a tempdir but does not clear JCODE_PROVIDER_PROFILE_NAME, JCODE_NAMED_PROVIDER_PROFILE, and JCODE_PROVIDER_PROFILE_ACTIVE env vars. When these are set in the user's shell (e.g. JCODE_PROVIDER_PROFILE_NAME=llmg-coding), init_provider_for_validation picks them up and fails with Unknown provider profile 'llmg-coding'.

3. init_provider_for_ollama_reapplies_local_compat_runtime_env_after_disabling_subscription_mode

Root cause: Same env var leak as #2. The test does not clear JCODE_PROVIDER_PROFILE_* env vars.

4. test_init_provider_jcode_delegates_runtime_profile_to_wrapper

Root cause: Same env var leak + the provider display name was changed from "Jcode Hosted Models" to "Jcode Subscription" but the test assertion was never updated. The test also does not clear JCODE_OPENROUTER_* env vars.

5. auto_provider_noninteractive_skips_untrusted_external_auth_instead_of_blocking

Root cause: The test does not clear JCODE_PROVIDER_PROFILE_* or JCODE_OPENROUTER_* env vars. In environments with JCODE_OPENROUTER_ALLOW_NO_AUTH=1 and JCODE_OPENROUTER_API_BASE set, OpenRouter initializes without credentials, causing the test to get a provider instead of the expected "No credentials configured" error.

Fix

A fix branch is available at: https://github.com/alecuba16/jcode/tree/fix/master-test-env-isolation

The fix:

  1. Updates the stale assertion in run_auto_poke_followup_targets_below_threshold_todos ("completion confidence" → "Validate further:")
  2. Adds JCODE_PROVIDER_PROFILE_NAME, JCODE_NAMED_PROVIDER_PROFILE, JCODE_PROVIDER_PROFILE_ACTIVE to the env var cleanup list in 4 provider init tests
  3. Adds JCODE_OPENROUTER_* env vars to the cleanup list in 2 tests that are affected by OpenRouter env var leaks
  4. Adds early invalidate_config_cache() calls after setting JCODE_HOME to ensure the config reloads from the tempdir
  5. Updates the provider display name assertion from "Jcode Hosted Models" to "Jcode Subscription"

After the fix, 4 of 5 tests pass. The remaining auto_provider_noninteractive failure is a deeper environment issue that requires further investigation into how register_external_provider_runtimes interacts with the OpenRouter env vars.

Environment

jcode v0.75.5-dev (2b49ed7b0)
macOS aarch64

The failures are reproducible when the shell has:

JCODE_PROVIDER_PROFILE_NAME=llmg-coding
JCODE_NAMED_PROVIDER_PROFILE=llmg-coding
JCODE_PROVIDER_PROFILE_ACTIVE=1
JCODE_OPENROUTER_ALLOW_NO_AUTH=1
JCODE_OPENROUTER_API_BASE=https://...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the five named failing tests with the listed provider-profile and OpenRouter environment variables set, then compare their environment cleanup with the fix/master-test-env-isolation branch. Confirm the stale assertions and isolation fixes, and investigate how register_external_provider_runtimes still affects auto_provider_noninteractive. Done means all five tests pass in the affected environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.