alibaba / alibaba/loongsuite-python

ci: crewai instrumentation tests failing on main (VCR cassette + OpenAI API connection error)

Open
#230 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
133
Forks
35
Avg merge
1d 12h
Merged PRs (30d)
5

Description

## Problem

The `loongsuite-instrumentation-crewai` test suite is failing on the `main` branch. This is a **pre-existing issue on main**, not introduced by any specific PR.

## Failing CI Runs

| Run | Branch | Conclusion | Date |
|-----|--------|------------|------|
| [#28212953288](https://github.com/alibaba/loongsuite-python/actions/runs/28212953288) | main | ❌ failure | 2026-06-26 02:18 UTC |
| [#28212331441](https://github.com/alibaba/loongsuite-python/actions/runs/28212331441) | main | ❌ failure | 2026-06-26 02:00 UTC |

Last passing run: [#28080659127](https://github.com/alibaba/loongsuite-python/actions/runs/28080659127) on 2026-06-24 06:49 UTC ✅

## Affected Versions

All Python versions: 3.10, 3.11, 3.12, 3.13 (Ubuntu)

## Failure Details

**1 failed, 47 passed** — the failing test is:

```
FAILED instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test_error_scenarios.py::TestErrorScenarios::test_api_key_missing
```

### Error chain:

1. **VCR cassette overwrite failure:**
```
WARNING LiteLLM: Failed to fetch remote model cost map from
https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json:
Can't overwrite existing cassette
(.../tests/cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml)
in your current record mode ().
```

2. **VCR matcher failures:**
```
Matchers failed:
method - assertion failure
host - assertion failure
path - assertion failure
```

3. **OpenAI API connection error (fallback to live API):**
```
ERROR root:completion.py:1772 Failed to connect to OpenAI API: Connection error.
ERROR root:completion.py:452 OpenAI API call failed: Failed to connect to OpenAI API: Connection error.
ERROR crewai.flow.runtime:__init__.py:2978 Error executing listener call_llm_and_parse: Failed to connect to OpenAI API: Connection error.
```

## Root Cause Hypothesis

The VCR cassette for `test_agent_workflow.TestAgentWorkflow.test_hierarchical_workflow` no longer matches the HTTP requests being made. When the cassette match fails, the test falls back to making a real OpenAI API call, which fails in CI (no network access to OpenAI API).

This likely started after one of these commits merged between 2026-06-24 and 2026-06-26:
- `1b25fcc5` feat(deepagents): support DeepAgents instrumentation (#228)
- `e4232fdc` feat(claude-agent-sdk): capture gen_ai.skill.* on Skill load execute_tool span (#226)

A dependency update (likely `crewai` or `litellm`) may have changed the HTTP request format, causing the cassette matchers (method/host/path) to fail.

## Suggested Fix

1. **Re-record the VCR cassette** for `test_hierarchical_workflow` with the current dependency versions
2. **Or** update the cassette matchers to be more lenient (e.g., match on `uri` instead of `method + host + path`)
3. **Or** pin the crewai/litellm version to match the cassette recording

## Impact

- Blocks PRs that trigger the crewai test matrix (e.g., [#229](https://github.com/alibaba/loongsuite-python/pull/229))
- The `test_api_key_missing` test in `test_error_scenarios.py` is also failing as a side effect

---
*Reported by github-manager-bot*

Contributor guide

Open the contributing guide

Research direction

Start by running instrumentation-loongsuite/loongsuite-instrumentation-crewai/tests/test_error_scenarios.py::TestErrorScenarios::test_api_key_missing and inspect tests/cassettes/test_agent_workflow/TestAgentWorkflow.test_hierarchical_workflow.yaml. Compare the cassette with the current requests and dependency versions, then verify the crewai test matrix passes without falling back to a live OpenAI API call.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.