5 pre-existing test failures on master (env var isolation + stale assertions)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 19.9k
- Forks
- 2.3k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 30
Description
Summary
5 tests fail on the current master branch (c4cdc6768) when run in an environment with provider profile and OpenRouter env vars set. These are environment isolation issues in the test code, not bugs in the application code.
Failing tests
1. run_auto_poke_followup_targets_below_threshold_todos
Root cause: Stale assertion. The test asserts message.contains("completion confidence") but the message format was changed to use "Validate further:" instead. The assertion was never updated.
2. explicit_anthropic_api_choice_pins_api_key_over_available_oauth
Root cause: The test sets JCODE_HOME to a tempdir but does not clear JCODE_PROVIDER_PROFILE_NAME, JCODE_NAMED_PROVIDER_PROFILE, and JCODE_PROVIDER_PROFILE_ACTIVE env vars. When these are set in the user's shell (e.g. JCODE_PROVIDER_PROFILE_NAME=llmg-coding), init_provider_for_validation picks them up and fails with Unknown provider profile 'llmg-coding'.
3. init_provider_for_ollama_reapplies_local_compat_runtime_env_after_disabling_subscription_mode
Root cause: Same env var leak as #2. The test does not clear JCODE_PROVIDER_PROFILE_* env vars.
4. test_init_provider_jcode_delegates_runtime_profile_to_wrapper
Root cause: Same env var leak + the provider display name was changed from "Jcode Hosted Models" to "Jcode Subscription" but the test assertion was never updated. The test also does not clear JCODE_OPENROUTER_* env vars.
5. auto_provider_noninteractive_skips_untrusted_external_auth_instead_of_blocking
Root cause: The test does not clear JCODE_PROVIDER_PROFILE_* or JCODE_OPENROUTER_* env vars. In environments with JCODE_OPENROUTER_ALLOW_NO_AUTH=1 and JCODE_OPENROUTER_API_BASE set, OpenRouter initializes without credentials, causing the test to get a provider instead of the expected "No credentials configured" error.
Fix
A fix branch is available at: https://github.com/alecuba16/jcode/tree/fix/master-test-env-isolation
The fix:
- Updates the stale assertion in
run_auto_poke_followup_targets_below_threshold_todos("completion confidence" → "Validate further:") - Adds
JCODE_PROVIDER_PROFILE_NAME,JCODE_NAMED_PROVIDER_PROFILE,JCODE_PROVIDER_PROFILE_ACTIVEto the env var cleanup list in 4 provider init tests - Adds
JCODE_OPENROUTER_*env vars to the cleanup list in 2 tests that are affected by OpenRouter env var leaks - Adds early
invalidate_config_cache()calls after settingJCODE_HOMEto ensure the config reloads from the tempdir - Updates the provider display name assertion from "Jcode Hosted Models" to "Jcode Subscription"
After the fix, 4 of 5 tests pass. The remaining auto_provider_noninteractive failure is a deeper environment issue that requires further investigation into how register_external_provider_runtimes interacts with the OpenRouter env vars.
Environment
jcode v0.75.5-dev (2b49ed7b0)
macOS aarch64
The failures are reproducible when the shell has:
JCODE_PROVIDER_PROFILE_NAME=llmg-coding
JCODE_NAMED_PROVIDER_PROFILE=llmg-coding
JCODE_PROVIDER_PROFILE_ACTIVE=1
JCODE_OPENROUTER_ALLOW_NO_AUTH=1
JCODE_OPENROUTER_API_BASE=https://...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the five named failing tests with the listed provider-profile and OpenRouter environment variables set, then compare their environment cleanup with the fix/master-test-env-isolation branch. Confirm the stale assertions and isolation fixes, and investigate how register_external_provider_runtimes still affects auto_provider_noninteractive. Done means all five tests pass in the affected environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100