Nothing refuses a comma in a doctest TEST_CASE name, and 1202 of them are already unselectable by -tc
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
Owed by .agents/specs/unaligned-safetensors-consumers.md under ## Owed, added in #2602.
What is unguarded
doctest splits its -tc= filter on commas, so a TEST_CASE name containing one can never be selected by that filter. The binary reports test cases: 0, assertions: 0 and Status: SUCCESS! at rc 0 -- a skip wearing a pass for anyone running a focused gate by name.
#2601 is one instance of this, repaired by renaming a single case in #2602. Nothing stops the next one. Measured on tests/ at 872f7bfe4:
$ grep -rhcE 'TEST_CASE\("[^"]*,[^"]*"\)' tests/ | paste -sd+ | bc
1202
Four of those sit in the two files #2581 touched, so the shape is ordinary here and not a slip somebody made once.
Evidence that the mechanism is exactly as described
Reproduced against doctest 2.5.2 on a pre-existing case, so the finding does not depend on the case #2601 renamed:
$ ./test_exl3_gemm -tc="exl3 policy: the shape table is upstream's, value for value"
[doctest] test cases: 0 | 0 passed | 0 failed | 15 skipped
[doctest] assertions: 0 | 0 passed | 0 failed |
[doctest] Status: SUCCESS! rc 0
The same case reached through a * glob selects 1 case and 25 assertions, and -tc="nomatchXYZ,exl3 policy: shape compatibility and the empty-block clamp" selects 1 case and 7 assertions. The filter therefore splits on the comma into independently full-matched patterns, and adding a space after the comma drops the selection back to 0.
Fix shape
A checker that refuses a comma inside a TEST_CASE (and SUBCASE) name, wired into scripts/agent-preflight.sh beside the other test-shape gates, with a red-before test. It must be diff-scoped or baselined: 1202 existing names would fail it on day one, and a gate that reds ordinary work is the defect rather than the discipline.
Renaming the 1202 is explicitly NOT what this issue asks for. The value is in stopping the 1203rd, which is written by somebody who does not know doctest does this.
Provenance
Raised as the durable follow-up to #2601 by the fresh review of #2602, which reported that the follow-up had no owner. It is filed rather than fixed here because a new checker changes gate semantics and wants its own spec, a red-before test, and a decision about the baseline -- none of which belong in a one-line rename.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read .agents/specs/unaligned-safetensors-consumers.md and scripts/agent-preflight.sh, then inspect the existing test-shape gates and the comma-name matches under tests/. Add a checker with a red-before test that catches new comma-containing TEST_CASE and SUBCASE names while preserving the existing baseline or diff scope. Done means ordinary existing names do not fail the gate, but the next invalid name does.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, cpp
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100