repo(tests): Reduce number of parameterized tests
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
One of the development bottlenecks is slow tests (locally and CI). In particular, there's an explosion of cases with parameterization as we add features. Some common parameters in the test suite:
- backend: duckdb, postgres, filesystem, etc.
- backend flavor: duckdb/motherduck, filesystem/iceberg, filesystem/s3
- file loader format: jsonl, csv, parquet
- dlt data type: bigint, float, timestamp
- source type: source, resource within a source, standalone resource
- reflection level: minimal, full, full with precision
- incremental configuration (multiple params): row order, last value function
- ... and many more
It would be highly valuable to manually review our entire test suite and reduce parameterization. This would involve writing a few more separate test that checks the condition. For example, `test_sql_database_source.py::test_reflect_foreign_keys_as_table` probably shouldn't be parameterized on `backend` (all assertions are about the dlt schema). Expected differences between `backend` should be tested explicitly in their own tests
Contributor guide
Research direction
Start with test_sql_database_source.py::test_reflect_foreign_keys_as_table and inspect which parameterized cases affect its assertions. Review the broader test suite for parameters that do not change the behavior under test; done means redundant parameterization is reduced and backend-specific differences have explicit tests where needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100