feast-dev / feast-dev/feast

Restructure test suite into 4-tier directory layout: unit, universal, component and integration

Open
#6,048 6 comments 2 reactions 0 assignees View on GitHub
good first issue kind/feature size/XL starter-ticket
Dominant language
Python
Stars
7.3k
Forks
1.4k
Avg merge
1d 21h
Merged PRs (30d)
15

Description

**Is your feature request related to a problem? Please describe.**

Our current test organization relies on pytest markers to separate unit and integration tests, but this system is failing:

- `test-python-unit` runs `pytest sdk/python/tests` (the **entire** tests directory) and filters by markers — any integration test missing `@pytest.mark.integration` silently runs as a "unit" test
- At least 3 integration test files (`test_mcp_feature_server.py`, `test_hybrid_online_store.py`, `test_dbt_integration.py`) are missing the marker and currently run during unit CI
- Unit tests import heavyweight dependencies (ray, pyspark) that slow down the entire suite and require special environment setup
- The `conftest.py` marker-filtering logic in `pytest_collection_modifyitems` is a house of cards — markers are opt-in and people forget them

**Describe the solution you'd like**

Replace marker-based filtering with a **folder-based** 4-tier layout:

```
tests/
unit/ → Fast, no heavy deps, pure logic
universal/ → Cross-backend parameterized tests (same test, many backends)
component/ → Backend-specific tests (needs ray/spark/etc installed)
integration/ → Needs external services (K8s, Docker, Keycloak, cloud)
```

*Key benefits:**
- `pytest tests/unit` — path-based, no marker gymnastics, no leakage
- `pytest tests/component/ray` — runs only ray-specific tests
- `pytest tests/universal --config=duckdb` — runs universal tests against DuckDB
- The folder **is** the categorization — no markers to forget
- Ray CI = `component/ray` + `universal(ray config)`. Clean and composable.

### Sub-tasks

- [ ] Create directory skeleton and move `feature_repos/`
- [ ] Move ray tests to `component/ray/`
- [ ] Move spark tests to `component/spark/`
- [ ] Move universal offline store tests to `universal/offline_store/`
- [ ] Move universal online store tests to `universal/online_store/`
- [ ] Move universal registration and materialization tests to `universal/`
- [ ] Reclassify misplaced tests (unit ↔ integration boundary fixes)
- [ ] Refactor root `conftest.py` — move fixtures, remove marker filtering
- [ ] Update Makefile targets for new directory layout
- [ ] Update CI workflows for new directory layout

### Migration Principles

1. Each sub-task is a self-contained PR that can be merged and tested independently
2. No functional changes to tests — only file moves and import path updates
3. Old Makefile targets should remain as aliases during transition (deprecated, then removed)
4. Run the relevant CI after each PR to verify nothing broke

**Thanks to @tokoko for idea and kicking off discussion.**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.