alibaba / alibaba/loongsuite-python
ci: crewai instrumentation tests failing on main (VCR cassette + OpenAI API connection error)
- Dominant language
- Python
- Stars
- 133
- Forks
- 35
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 5
Description
## Problem
The `loongsuite-instrumentation-crewai` test suite is failing on the `main` branch. This is a **pre-existing issue on main**, not introduced by any specific PR.
## Failing CI Runs
| Run | Branch | Conclusion | Date |
|-----|--------|------------|------|
| [#28212953288](https://github.com/alibaba/loongsuite-python/actions/runs/28212953288) | main | ❌ failure | 2026-06-26 02:18 UTC |
| [#28212331441](https://github.com/alibaba/loongsuite-python/actions/runs/28212331441) | main | ❌ failure | 2026-06-26 02:00 UTC |
Last passing run: [#28080659127](https://github.com/alibaba/loongsuite-python/actions/runs/28080659127) on 2026-06-24 06:49 UTC ✅
## Affected Versions
All Python versions: 3.10, 3.11, 3.12, 3.13 (Ubuntu)
## Failure Details
**1 failed, 47 passed** — the failing test is:
```
FAILED instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test_error_scenarios.py::TestErrorScenarios::test_api_key_missing
```
### Error chain:
1. **VCR cassette overwrite failure:**
```
WARNING LiteLLM: Failed to fetch remote model cost map from
https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json:
Can't overwrite existing cassette
(.../tests/cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml)
in your current record mode ().
```
2. **VCR matcher failures:**
```
Matchers failed:
method - assertion failure
host - assertion failure
path - assertion failure
```
3. **OpenAI API connection error (fallback to live API):**
```
ERROR root:completion.py:1772 Failed to connect to OpenAI API: Connection error.
ERROR root:completion.py:452 OpenAI API call failed: Failed to connect to OpenAI API: Connection error.
ERROR crewai.flow.runtime:__init__.py:2978 Error executing listener call_llm_and_parse: Failed to connect to OpenAI API: Connection error.
```
## Root Cause Hypothesis
The VCR cassette for `test_agent_workflow.TestAgentWorkflow.test_hierarchical_workflow` no longer matches the HTTP requests being made. When the cassette match fails, the test falls back to making a real OpenAI API call, which fails in CI (no network access to OpenAI API).
This likely started after one of these commits merged between 2026-06-24 and 2026-06-26:
- `1b25fcc5` feat(deepagents): support DeepAgents instrumentation (#228)
- `e4232fdc` feat(claude-agent-sdk): capture gen_ai.skill.* on Skill load execute_tool span (#226)
A dependency update (likely `crewai` or `litellm`) may have changed the HTTP request format, causing the cassette matchers (method/host/path) to fail.
## Suggested Fix
1. **Re-record the VCR cassette** for `test_hierarchical_workflow` with the current dependency versions
2. **Or** update the cassette matchers to be more lenient (e.g., match on `uri` instead of `method + host + path`)
3. **Or** pin the crewai/litellm version to match the cassette recording
## Impact
- Blocks PRs that trigger the crewai test matrix (e.g., [#229](https://github.com/alibaba/loongsuite-python/pull/229))
- The `test_api_key_missing` test in `test_error_scenarios.py` is also failing as a side effect
---
*Reported by github-manager-bot*
Contributor guide
Research direction
Start by running instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test_error_scenarios.py::TestErrorScenarios::test_api_key_missing and inspect tests/cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml. Compare the cassette with the current requests and dependency versions, then verify the crewai test matrix passes without falling back to a live OpenAI API call.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100