ContextLab / ContextLab/orchestrator

Bring the legacy test suite to green (~540 failures outside the blocking layer)

Open
#354 7 comments 0 reactions 1 assignee Claimed by @jeremymanning View on GitHub
bug epic high-priority
Dominant language
Python
Stars
3
Forks
2
Avg merge
13m
Merged PRs (30d)
1

Description

## Overview

Following the major refactor in #307, we need to systematically audit our entire test suite to achieve a 100% pass rate while using tests to guide development.

## Current Status
- 108 test files have been updated with correct import paths (orchestrator. → src.orchestrator.)
- Many tests may be obsolete due to architectural changes in refactor #307
- Some tests are broken but should work with current architecture
- Failing tests should guide toolbox improvements and missing functionality

## Systematic Approach

### Phase 1: Test Categorization
- [ ] Run complete test suite and capture all failures
- [ ] Categorize each failing test as:
- **OBSOLETE**: No longer relevant after refactor, should be removed
- **BROKEN**: Should work with current architecture, needs repair
- **GUIDE**: Reveals missing functionality that needs implementation

### Phase 2: Test Cleanup
- [ ] Remove all OBSOLETE tests that no longer apply to current architecture
- [ ] Document removed tests and reasoning

### Phase 3: Test Repair
- [ ] Fix all BROKEN tests to work with current architecture
- [ ] Update test assertions to match new API contracts
- [ ] Ensure tests use correct import paths and class structures

### Phase 4: Test-Driven Development
- [ ] Use GUIDE tests to implement missing functionality
- [ ] Create new GitHub issues for large-scale implementations
- [ ] Fix small missing pieces immediately

### Phase 5: Validation
- [ ] Achieve 100% test pass rate locally
- [ ] Verify CI passes across all platforms (Ubuntu, macOS, Windows)
- [ ] Verify CI passes across all Python versions (3.11, 3.12)

## Success Criteria
- 100% test pass rate locally and in CI
- All obsolete tests removed and documented
- All broken tests repaired or converted to implementation guides
- Test suite guides future development effectively

## Related Issues
- Closes: orchestrator-api-completion epic
- Related: #307 (major refactor)
- Builds on: Import path fixes across 108 test files

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.