garrytan / garrytan/gstack

/retro test-file count uses a JS-only glob, undercounts Python repos (and overcounts if naively fixed)

Open
#1,999 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133k
Forks
19.9k
Avg merge
18h 46m
Merged PRs (30d)
26

Description

## Problem

`/retro` Step 1 command 10 counts repository test files with a JavaScript/TypeScript-shaped glob:

`find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l`

Python's dominant convention is `test_*.py`, which none of those four patterns match. On a Python repo the `test_health.total_test_files` metric silently lands on whatever stray `*_test.*` files happen to exist, never the real count. Because Step 13 persists this number and Step 12 trends it, the test-health trend line measures noise, not test growth.

There's a second failure mode hiding behind the obvious fix. Naively adding `test_*.py` over-corrects, because the only directory excluded is `node_modules`, while Python projects keep their dependencies in a virtualenv (`.venv/`, `venv/`, `site-packages/`) full of third-party test suites.

Measured on a real Python repo this week:

| Method | Count | Why |
|--------|-------|-----|
| current gstack glob | 54 | misses `test_*.py` entirely |
| naive `find . -name 'test_*.py'` | 1,496 | 1,188 of those live in `.venv/` (dependency tests) |
| correct project count (`tests/`) | 297 | the actual project test files |

So the live metric reported 54 where the truth is 297, a 5.5x undercount.

## Proposed fix

Two parts, both required:

1. Extend the glob with common non-JS conventions: `test_*.py`, `*_test.py`, `*_test.go`, `*_test.rb`, `*Test.java`, `*_spec.rb`. The existing JS/TS patterns stay.
2. Exclude virtualenv and vendored dependency dirs, not just `node_modules`: add `.venv`, `venv`, any `*-venv`, and `site-packages` to the prune/grep filter. Otherwise the Python fix sweeps in thousands of dependency test files.

A `-prune`-based find (or a `grep -vE '(/node_modules/|/\.venv/|/venv/|/site-packages/)'` filter) handles both the count (command 10) and the test-files-changed metric (command 12) consistently.

## Why it matters

`test_health` is one of the few quality signals the retro trends over time. On every Python repo it's been reporting a number 5x too low, so "tests added this period" deltas are meaningless. The fix is a few characters in two `find` commands and makes the metric correct for the largest non-JS ecosystem.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.