/retro test-file count uses a JS-only glob, undercounts Python repos (and overcounts if naively fixed)
- Dominant language
- TypeScript
- Stars
- 133k
- Forks
- 19.9k
- Avg merge
- 18h 46m
- Merged PRs (30d)
- 26
Description
## Problem
`/retro` Step 1 command 10 counts repository test files with a JavaScript/TypeScript-shaped glob:
`find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l`
Python's dominant convention is `test_*.py`, which none of those four patterns match. On a Python repo the `test_health.total_test_files` metric silently lands on whatever stray `*_test.*` files happen to exist, never the real count. Because Step 13 persists this number and Step 12 trends it, the test-health trend line measures noise, not test growth.
There's a second failure mode hiding behind the obvious fix. Naively adding `test_*.py` over-corrects, because the only directory excluded is `node_modules`, while Python projects keep their dependencies in a virtualenv (`.venv/`, `venv/`, `site-packages/`) full of third-party test suites.
Measured on a real Python repo this week:
| Method | Count | Why |
|--------|-------|-----|
| current gstack glob | 54 | misses `test_*.py` entirely |
| naive `find . -name 'test_*.py'` | 1,496 | 1,188 of those live in `.venv/` (dependency tests) |
| correct project count (`tests/`) | 297 | the actual project test files |
So the live metric reported 54 where the truth is 297, a 5.5x undercount.
## Proposed fix
Two parts, both required:
1. Extend the glob with common non-JS conventions: `test_*.py`, `*_test.py`, `*_test.go`, `*_test.rb`, `*Test.java`, `*_spec.rb`. The existing JS/TS patterns stay.
2. Exclude virtualenv and vendored dependency dirs, not just `node_modules`: add `.venv`, `venv`, any `*-venv`, and `site-packages` to the prune/grep filter. Otherwise the Python fix sweeps in thousands of dependency test files.
A `-prune`-based find (or a `grep -vE '(/node_modules/|/\.venv/|/venv/|/site-packages/)'` filter) handles both the count (command 10) and the test-files-changed metric (command 12) consistently.
## Why it matters
`test_health` is one of the few quality signals the retro trends over time. On every Python repo it's been reporting a number 5x too low, so "tests added this period" deltas are meaningless. The fix is a few characters in two `find` commands and makes the metric correct for the largest non-JS ecosystem.
Contributor guide
Assessment
This issue has not been assessed yet.