bcgov / bcgov/lcfs

LCFS - Separate Backend Unit and Integration Tests with Parallelization Support

Open
#2,972 0 comments 0 reactions 0 assignees View on GitHub
Low Tech Debt
Dominant language
Python
Stars
6
Forks
6
Avg merge
2d 4h
Merged PRs (30d)
87

Description

## Describe the task

Refactor the backend test suite to clearly separate unit tests from integration tests, enabling:
1. Parallel execution of unit tests using pytest-xdist for faster CI/CD pipelines
2. Clear distinction between tests that require database/external dependencies and those that don't
3. Potential future parallelization of integration tests through database isolation strategies

Currently, our backend tests are mixed together without clear categorization. All tests run sequentially, creating a bottleneck in our CI/CD pipeline. The frontend tests already use sharding (8 parallel jobs), demonstrating the performance benefits of parallelization.

## Purpose

**Why this task is needed:**
- **Performance**: Backend tests currently run sequentially, taking significant time in CI/CD. The frontend already benefits from 8-way parallelization.
- **Developer Experience**: Faster test execution enables quicker feedback loops during development
- **Scalability**: As our test suite grows, sequential execution becomes increasingly problematic
- **Resource Optimization**: Unit tests don't need database setup, allowing them to run in lightweight containers
- **Test Organization**: Clear separation improves maintainability and helps developers understand test dependencies

**Value it adds:**
- Reduce CI/CD pipeline time by potentially 50-70% for unit tests
- Enable developers to run fast unit tests frequently during development
- Provide foundation for more advanced testing strategies (contract testing, property-based testing)
- Align with testing best practices and make onboarding easier

## Acceptance Criteria

### Phase 1: Test Categorization and Markers
- [ ] Add pytest markers to categorize tests:
```python
@pytest.mark.unit # Pure unit tests with mocked dependencies
@pytest.mark.integration # Tests requiring database/Redis/external services
@pytest.mark.slow # Tests known to take >1s
```
- [ ] Update `pyproject.toml` with marker definitions and documentation
- [ ] Audit all existing tests in `/backend/lcfs/tests/` and categorize them appropriately
- [ ] Create documentation explaining the criteria for each test category

### Phase 2: Unit Test Isolation
- [ ] Refactor unit tests to use mocks/fakes instead of real database connections
- [ ] Create additional mock fixtures for common dependencies (repositories, services)
- [ ] Ensure unit tests can run without Docker containers or database setup
- [ ] Verify unit tests run in <100ms each

### Phase 3: Parallel Execution for Unit Tests
- [ ] Add `pytest-xdist` to project dependencies
- [ ] Configure pytest to run unit tests in parallel (e.g., `pytest -n auto`)
- [ ] Update CI/CD workflow to run unit tests with parallelization
- [ ] Add separate CI job for unit tests that runs before integration tests
- [ ] Measure and document performance improvements

### Phase 4: Integration Test Optimization
- [ ] Investigate database isolation strategies for parallel integration tests:
- Option 1: Multiple test databases with unique names per worker
- Option 2: Schema-based isolation (PostgreSQL schemas)
- Option 3: Transaction-based isolation with pytest-xdist coordination
- [ ] Implement proof-of-concept for chosen strategy
- [ ] Document any limitations or trade-offs

### Phase 5: CI/CD Pipeline Updates
- [ ] Update `docker-auto-test.yaml` to run tests in stages:
1. Parallel unit tests (fail fast)
2. Integration tests (potentially parallel)
3. Coverage report aggregation
- [ ] Add test timing reports to identify slow tests
- [ ] Configure test result caching for unchanged code
- [ ] Update documentation and developer guides

## Additional Context

### Current State Analysis
- **Test Count**: Approximately 200+ test files across various modules
- **Current Execution**: All tests run sequentially with `poetry run pytest`
- **Database Setup**: Single `lcfs_test` database created/dropped for entire test run
- **Frontend Comparison**: Frontend tests already use 8-way sharding, demonstrating feasibility

### Technical Considerations
1. **pytest-xdist**: Industry standard for pytest parallelization
2. **Database Isolation**: PostgreSQL supports multiple strategies for parallel testing
3. **Redis Mocking**: Already using `fakeredis` for cache testing
4. **Authentication**: Mock authentication system already in place

### Example Test Categorization
```python
# Unit test example (no database needed)
@pytest.mark.unit
async def test_calculate_compliance_units():
# Pure business logic testing
result = calculate_units(100, 0.5)
assert result == 50

# Integration test example (needs database)
@pytest.mark.integration
@pytest.mark.anyio
async def test_create_compliance_report(dbsession):
# Test with real database interaction
report = await create_report(dbsession, test_data)
assert report.id is not None
```

### Success Metrics
- Unit test execution time: <30 seconds for entire suite
- Integration test execution time: <50% reduction if parallelized
- Overall CI/CD time: 40-60% reduction
- Zero flaky tests due to parallelization issues

### References
- Frontend sharding implementation: `.github/workflows/docker-auto-test.yaml:112-132`
- Current pytest configuration: `backend/pyproject.toml:124-131`
- Test fixtures: `backend/lcfs/conftest.py`

Contributor guide

Open the contributing guide

Research direction

Start by reading backend/pyproject.toml, backend/lcfs/conftest.py, the tests under backend/lcfs/tests/, and .github/workflows/docker-auto-test.yaml:112-132; run the current suite with poetry run pytest. Done means the agreed test categories, isolation approach, dependency configuration, CI stages, timing reports, and documentation are implemented and verified without parallelization flakiness.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, postgresql, python, redis
Domain
ci-cd, databases, devops, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.