microsoft / microsoft/agent-framework

Python: dependency-bounds validator is not enforced in CI

Open
#6,582 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

harness python
Dominant language
Python
Stars
13.6k
Forks
2.3k
Avg merge
2d 45m
Merged PRs (30d)
358

Description

The Python dependency-bounds validator (`uv run poe validate-dependency-bounds-test`) is not wired into required CI, so dependency-floor, packaging, and typing regressions can land on main and ship in a release without being caught.

Evidence

Running the validator at the `python-1.8.1` release tag already fails for core at the `lowest-direct` resolution:

```
packages/core: lower failed
Task 'test' failed for packages/core at resolution 'lowest-direct'.
FAILED tests/core/test_telemetry.py::test_detect_hosted_fallback_import_error
1 failed, 3174 passed, 18 skipped, 2 xfailed
```

So 1.8.1 was tagged and released while the validator was red.

On current main the same validator fails with more: the telemetry test above, plus 4 harness shell-tool tests (`ModuleNotFoundError: No module named 'agent_framework_tools'`) and 14 pyright errors on `_harness/_agent.py`, all from the shell-tool integration in #6451. The validator stops at the first failing task, which is why only the telemetry failure surfaced at 1.8.1.

Two latent problems this masked

1. `packages/core/agent_framework/_telemetry.py` caught `(ModuleNotFoundError, ValueError)` around `importlib.util.find_spec(...)`, but `find_spec` imports the parent package and can raise a bare `ImportError`, which escaped the handler. The test passes in the full test environment (where `azure.ai.agentserver` is already imported, so `find_spec` reuses the cached parent and never hits the mocked import) but fails in the validator's isolated environment. Pre-existing since #5455.

2. The harness shell-tool integration (#6451) imports `agent_framework_tools` lazily because that package depends on core, so core cannot take a runtime dependency on it. But core did not declare it even as a dev or optional dependency, so any environment that builds core in isolation hits `ModuleNotFoundError` at test time and unresolved types under pyright.

Both are fixed in the 1.9.0 version bump (branch `bump-py-ver-release-260617`): broaden the except to `ImportError`, add an `agent-framework-tools` dev dependency-group to core, and guard the shell-tool tests with `skipif`. With those, the validator passes fully (lower and upper, all packages).

Proposal

Wire `validate-dependency-bounds-test` into Python CI so these regressions are caught at PR time rather than at release time:

- Run it on PRs that touch `python/`, or at minimum on the release branch and nightly.
- It exercises both `lowest-direct` and `highest` resolutions per package and runs each package's pyright pass, so it catches floor-too-low, missing-optional-dependency, and type-checking-against-isolated-env problems that the normal full-workspace test job does not.

Cost and scoping note: a full workspace run is somewhat slow (lower plus upper per package). If per-PR cost is a concern, options are to scope the run to changed packages, or run the full sweep nightly plus required-on-release-branch.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the existing Python CI configuration and the `uv run poe validate-dependency-bounds-test` entry point first. Determine how required PR, release-branch, or nightly jobs are defined and whether package scoping is supported. Done means the validator runs in the chosen CI scope and fails the job when dependency-bound or pyright checks fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.