mudler / mudler/vllm.cpp

Nothing refuses a comma in a doctest TEST_CASE name, and 1202 of them are already unselectable by -tc

Open
#2,605 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owed by .agents/specs/unaligned-safetensors-consumers.md under ## Owed, added in #2602.

What is unguarded

doctest splits its -tc= filter on commas, so a TEST_CASE name containing one can never be selected by that filter. The binary reports test cases: 0, assertions: 0 and Status: SUCCESS! at rc 0 -- a skip wearing a pass for anyone running a focused gate by name.

#2601 is one instance of this, repaired by renaming a single case in #2602. Nothing stops the next one. Measured on tests/ at 872f7bfe4:

$ grep -rhcE 'TEST_CASE\("[^"]*,[^"]*"\)' tests/ | paste -sd+ | bc
1202

Four of those sit in the two files #2581 touched, so the shape is ordinary here and not a slip somebody made once.

Evidence that the mechanism is exactly as described

Reproduced against doctest 2.5.2 on a pre-existing case, so the finding does not depend on the case #2601 renamed:

$ ./test_exl3_gemm -tc="exl3 policy: the shape table is upstream's, value for value"
[doctest] test cases: 0 | 0 passed | 0 failed | 15 skipped
[doctest] assertions: 0 | 0 passed | 0 failed |
[doctest] Status: SUCCESS!                                        rc 0

The same case reached through a * glob selects 1 case and 25 assertions, and -tc="nomatchXYZ,exl3 policy: shape compatibility and the empty-block clamp" selects 1 case and 7 assertions. The filter therefore splits on the comma into independently full-matched patterns, and adding a space after the comma drops the selection back to 0.

Fix shape

A checker that refuses a comma inside a TEST_CASE (and SUBCASE) name, wired into scripts/agent-preflight.sh beside the other test-shape gates, with a red-before test. It must be diff-scoped or baselined: 1202 existing names would fail it on day one, and a gate that reds ordinary work is the defect rather than the discipline.

Renaming the 1202 is explicitly NOT what this issue asks for. The value is in stopping the 1203rd, which is written by somebody who does not know doctest does this.

Provenance

Raised as the durable follow-up to #2601 by the fresh review of #2602, which reported that the follow-up had no owner. It is filed rather than fixed here because a new checker changes gate semantics and wants its own spec, a red-before test, and a decision about the baseline -- none of which belong in a one-line rename.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read .agents/specs/unaligned-safetensors-consumers.md and scripts/agent-preflight.sh, then inspect the existing test-shape gates and the comma-name matches under tests/. Add a checker with a red-before test that catches new comma-containing TEST_CASE and SUBCASE names while preserving the existing baseline or diff scope. Done means ordinary existing names do not fail the gate, but the next invalid name does.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, cpp
Domain
testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.